Open view, multi-modal, calibrated digital loupe with depth sensing

ABSTRACT

A digital loupe system is provided which can include a number of features. In one embodiment, the digital loupe system can include a stereo camera pair and a distance sensor. The system can further include a processor configured to perform a transformation to image signals from the stereo camera pair based on a distance measurement from the distance sensor and from camera calibration information. In some examples, the system can use the depth information and the calibration information to correct for parallax between the cameras to provide a multi-channel image. Ergonomic head mounting systems are also provided. In some implementations, the head mounting systems can be configurable to support the weight of a digital loupe system, including placing one or two oculars in a line of sight with an eye of a user, while improving overall ergonomics, including peripheral vision, comfort, stability, and adjustability. Methods of use are also provided.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.17/156,191, filed Jan. 22, 2021, which application claims the benefit ofpriority of U.S. Provisional Application No. 62/964,287, filed Jan. 22,2020, entitled “DIGITAL LOUPE WITH CALIBRATED DEPTH SENSING”, both ofwhich are incorporated by reference as if fully set forth herein.

INCORPORATION BY REFERENCE

All publications and patent applications mentioned in this specificationare herein incorporated by reference in their entirety to the sameextent as if each individual publication or patent application wasspecifically and individually indicated to be incorporated by reference.

FIELD

This disclosure describes devices and methods for improving digitalmagnifying loupes. More specifically, these devices and methods allowfor an increased range of working distances, superior visual ergonomics,and the incorporation of advanced multi-channel optical imagingmodalities.

BACKGROUND

Surgeons, dentists, jewelers, and others whose work relies on precisehand-eye coordination at a miniature scale have long used binocularloupes as a visual aid. Such loupes comprise a pair of non-invertingtelescopes with a working distance of approximately 0.5 m, that is, thedistance from the eyes of the user to the nominal point of convergenceof the optical axes of the two telescopes, which in normal usage is thelocation of the subject or work area under observation, is approximately0.5 m. The telescopes are usually embedded in a user's spectacles in a“near-vision” position, similar to the near-vision position at thebottom of the lenses of bifocals, except they offer an angularmagnification of around 2× to 3× over a relatively limited field ofview, while permitting both peripheral and “far” vision when the userlooks around the telescopes.

The term “digital loupe” has been used to refer to loupe-like systems,often for use in surgery, where a focal plane array (image sensor) isplaced at the focal plane of each of the telescopes to digitize theimage. The digitized images can be transformed through various forms ofsignal processing before being displayed at the focal planes of twoeyepieces or oculars, one for each eye. This arrangement forms abinocular head-mounted display (HMD) with a digitally created magnifiedview of the work area.

With a digital loupe comes both many challenges and many opportunities.For example, there is the added weight of the image sensors, displays,and other electronic components, as well as the loss of depth of fieldthat would otherwise come from the eye's natural focusing accommodation.However, as will be explained in the context of the present disclosure,digital technology brings capabilities like image stabilization,automatic focus, and automatic convergence that enable magnificationsapproaching those of a surgical microscope. Such capabilities enableflexibility of working distance and freedom of movement, neither ofwhich are afforded by such microscopes, nor by traditional analogloupes. Furthermore, the bifurcation of the sensing/imaging side and thedisplay side of the loupes, enabled by the digital, rather than optical,information transfer between the two, allows for separate optimizationof their mounting configurations. As will be shown, this creates moreergonomic working conditions for the surgeon, such as a more verticalhead position, as well as the ability to simultaneously and concurrentlyview an object or work area directly (i.e., without looking through theloupes) and through the loupes. Finally, with digital technology, it ispossible to include more advanced optical imaging modalities such asfluorescence imaging or hyperspectral imaging, e.g., for a visualizationof tumor margins overlaid on the loupe image.

There are several outstanding challenges of digital loupes in the priorart that embodiments of the present disclosure aim to solve. First, withhigh-magnification binocular systems, a condition known as diplopia ordouble vision is known to arise, especially if left and right opticalaxes of the system are not properly aligned. Also, at highermagnifications, slight changes in working distance may translate tolarge relative shifts in the positions of left and right images, suchthat the human visual system cannot comfortably maintain single vision.The prior art has attempted to overcome this, but in an incompletemanner, whereas the present disclosure overcomes this challengecompletely by incorporating a distance sensor with a defined angularfield of view as well as a processor with camera calibration informationthat is used to electronically transform the image along withmeasurements from the distance sensor. We now review the prior artrelevant to this first outstanding challenge before delineating theothers.

It was recognized some time ago that, just as it is important for acamera to have autofocus to maintain a sharp image as the distance to asubject is changed, so a set of loupes should automatically adjust itshorizontal convergence angle, or the acute angle formed between theoptical axes of the left and right telescopes as viewed in a topprojection, such that the optical axes of the left and right telescopesconverge to the subject or work area being observed. U.S. Pat. No.5,374,820 teaches using a distance sensor on a traditional (analog)loupe to measure the distance to a subject. This distance measurement isthen used to mechanically change, in a corresponding fashion, the focaldistance and the convergence angle of the telescopes, or oculars.However, such mechanical movement is not sufficiently precise at highmagnification, there is no provision for incorporation of calibrationinformation that might be used to correct for angular misalignments(both horizontal and vertical) of the telescopes as a function ofdistance, and the distance sensor does not have a defined field of view.There is only a provision for adjusting the convergence angle as viewedin a top projection, that is, the horizontal convergence angle. The eyesare generally more sensitive to image misalignments in the verticaldirection, but this patent does not teach a method to overcome any suchmisalignments, which may be caused by slightly different tilts of theoculars relative to their as-designed or as-intended configuration.

WO2001005161 teaches a digital surgical loupe that dynamically optimizesthe image based on surgical conditions. It teaches that the optimalstereoscopic vision is given when a baseline (corresponding to theinterpupillary distance (IPD) of the stereo camera pair) is about 1/20thof the working distance. Based on the focus distance inferred from thebest focus setting of the stereo camera pair, this system has motorsthat first adjust the IPD to be in the optimal range, and thensubsequently adjust the horizontal convergence angle (in the plane ofthe two cameras and the subject) so the cameras of the stereo pairconverge on the subject. However, the use of focus setting as a proxyfor a true distance measurement to the subject is too inaccurate for theneeds of a high-magnification loupe—for example, conversion of a focussetting to a distance may be accurate to within a few cm, whereas adistance accuracy of better than a few mm is needed for an optimalsystem. Also, the use of motors to adjust IPD and convergence angleleads to a bulky system and may lack sufficient accuracy, repeatability,stability, and rapid settling to a given convergence angle setting.There is no provision to include camera calibration information thatcould be used to actively correct for horizontal and verticalmisalignments between a horizontal convergence angle setting and theactual camera orientations. The present disclosure overcomes theselimitations.

Some embodiments of digital loupes and/or augmented reality headsetswithin the prior art rely on methods that do not use distance sensors ordirect distance measurements to determine a convergence angle, whileothers also do not rely on motor-driven systems. For example, US2010/0045783 teaches, for a video see-through head-mounted display usedin a surgical context, a method of dynamic virtual convergence. Eachcamera of a stereo pair has a field of view larger than the displaysused to display the camera images, and a heuristic (for example, thedistance to the points closest to the viewer within an estimated scenegeometry, or the distance to a tracked tool) is used to estimate thegaze distance of the viewer. Display frustums are transformed,electronically, to match the estimated gaze distance. Effectively, theconvergence is virtually adjusted because only a portion of each cameraimage is shown on each corresponding display, that portion whichcorresponds to the object that the viewer is gazing at, and whichdepends on the distance to that object. One notable feature of thesystem described in US 2010/0045783 is the use of filtering hightemporal frequency components of the gaze distance. However, the user'sgaze distance is not accurately measured by an independent sensor. Also,the display frustums are transformed to a convergence angle of 0degrees, i.e., parallel vision, as if the object is at infinity, so thatthe technique can be used with conventional binocular head-mounteddisplays that have a relative ocular convergence angle of 0 degrees.This approach creates a vergence-disparity conflict, whereby thehorizontal disparities (pixel shifts) between the left and right imagesare the same as if the object were at its original (near) location, butthe lack of convergence of the eyes sends a conflicting signal to thebrain that the object is far away. Thus, this approach is not useful forcomfortably maintaining concurrence between peripheral near vision andaugmented vision, where one may want to switch between a magnified oraugmented view of an object or work area and a direct view of the objector work area while looking through the oculars or displays, and whilemaintaining the same vergence state of the eyes when looking over orunder the oculars or displays. The present disclosure overcomes thislimitation with an ocular convergence angle of the head-mounted displaythat nominally matches the actual working distance of the user and witha processor that can transform the images from the stereo camera pairsuch that the eyes do not substantially need to change their vergencestate when switching between viewing the images of an object or workarea through the head-mounted display and concurrently viewing theobject or work area directly over a range of working distances.

Some embodiments of digital loupes use position tracking or imagefeature tracking to maintain an object within the field of view of botheyes, effectively maintaining a convergence of a stereo camera pair tothat object. U.S. Pat. No. 9,967,475 B2 teaches a digital loupe thatrequires an operator to manually select an object of interest in animage, and a processor that determines the line of sight of the objectof interest relative to the camera optical axis and re-positions andcrops a portion of the image based on tracked head deviations from theline of sight, such that the object of interest stays within the centerof the cropped image. US 20160358327 A1 teaches a sort of digital loupe,wherein a live, magnified view of a (dental) work area is provided, andautomatically tracked to keep it centered within the field of view of ahead-mounted display, using image feature recognition and micro pan andtilt adjustments of the attached cameras. U.S. Pat. No. 9,690,119 B2teaches a first optical path and a second optical path on a head-wornapparatus where the direction of the first optical path is separatelyadjustable relative to the direction of the second optical path, and amagnified image passes between the two. U.S. Pat. No. 9,690,119 B2 alsoteaches the setting of a convergence angle (e.g., using adjustablemirrors) such that single vision occurs at the working distance, and itteaches automatic tracking of a point within the field of view byrecognition of implicit or explicit features, but it does not teach theuse of a direct measurement of distance to the subject.

U.S. Pat. No. 9,772,495 teaches a digital loupe that is intended toreplace both the conventional surgical loupe and the surgicalmicroscope. It teaches two cameras, each on an axial rotation system,and an illumination module. The axial rotation systems and illuminationmodule respond to a feedback signal derived from the two cameras tomaintain consistent illumination and a stable image. Moreover, whileU.S. Pat. No. 9,772,495 teaches that the axial rotation modules rotateto allow for capturing a desired surgical view, no provision is givenfor how to determine, either manually or automatically, what thisdesired view comprises, nor how to track it as the surgeon moves around.It also explains that the images from the two cameras have to be alignedto avoid double vision, presumably by rotating the cameras, but noexplanation or details are given about how this is done. In any case,embodiments that use position or image feature tracking count on theability to derive robust, precise, and reliable estimates of distance tothe subject, which these methods cannot give. For example, image featuretracking relies on the presence of distinct features in an image, whichcannot always be assumed due to the existence of relatively featurelessor textureless subjects.

A second challenge of digital loupes in the prior art that the presentdisclosure overcomes relates to the incorporation of multiple opticalimaging modalities. Some modalities, such as hyperspectral imaging,depend on the measurement of multiple channels for a given image point.There are various examples in the prior art of digital loupesincorporating advanced imaging modalities; however, it is known thatmulti-channel modalities like hyperspectral imaging may be hard tointegrate in digital loupes due to the bulk of the instruments and/or totradeoffs involving spatial or temporal resolution, etc. An aspect ofthe present disclosure is to form a hyperspectral imager or othermulti-channel imager (e.g., Stokes imaging polarimeter) that is smallenough to include in a digital loupe, yet without sacrificing lightthroughput or temporal or spatial resolution, by inclusion of an imagingdepth sensor and a calibrated array of single-channel imagers. Aprocessor uses a depth image from the imaging depth sensor to removeparallax from images from the single-channel imagers such that theyappear to have been captured from the same viewpoint, just like in moreconventional multi-channel yet single-viewpoint imagers. Within thepresent disclosure, “channel” may refer to an individual wavelengthband, an individual polarization component, or a corresponding notion;or, it may refer to an image acquired from light corresponding to one ofthese notions of channel. Thus, multispectral and hyperspectral imagersare multi-channel imagers because they image multiple wavelength bands,and a Stokes imaging polarimeter is a multi-channel imager because itimages multiple polarization components.

Previous embodiments of digital loupes have incorporated multipleoptical imaging modalities. For example, U.S. Pat. No. 6,032,070 teachesreflecting light off tissue and imaging, using various optical methods(different wavelength bands, polarizations, etc.) and digital processingto enhance contrast of tissue structures beyond what is visible with thenaked eye. This is done in conjunction with a helmet or head mounteddevice, such that the enhanced contrast image is displayedstereoscopically along the line of sight of the user. WO 2011002209 A2teaches a digital loupe that combines magnification and illumination invarious spectral bands, and has a manually-adjustable convergence of thecameras. WO 2018235088 A1 teaches a digital loupe with an array ofcameras for each eye with the same working distance, e.g., on aheadband. The different cameras within an array for a given eye may havedifferent magnifications, or may comprise a color camera and an infraredcamera, in such a way that at least two corresponding cameras from leftand right eyes are used to provide a stereoscopic view. A manualcontroller is used to select a low magnification stereoscopic image, ora high magnification stereoscopic image, or an infrared stereoscopicimage, etc. Note that while this publication discloses an array ofcameras, it does not teach fusing the images from the array into asingle viewpoint multi-channel image using spatially resolved distanceinformation as does the present disclosure. For the purposes of thepresent disclosure, a multi-channel imager may comprise single channelsat different magnifications.

U.S. Ser. No. 10/230,943 B2 teaches a type of digital loupe withintegrated fluorescence imaging such that within one sensor, both NIR(fluorescence) and visible light are recorded, with a modified Bayerpattern where pixels in both visible and infrared bands can be tiled onthe same sensor. This simplifies co-registration of color RGB and NIRfluorescence images because they are obtained from the same viewpoint.However, with current image sensor technology this technique is somewhatimpractical due to the significantly different optimal imagingconditions desired for each modality. Each modality may have differentoptimal exposure times, gains, resolutions, pixel sizes, etc., butbecause the modalities are being recorded on the same sensor, they mustbe recorded with the same conditions if they are to be recordedsimultaneously. There is also a loss of spatial resolution for eachmodality due to the sharing of the image sensor's pixels acrossmodalities.

US 2018/0270474 A1 teaches registration of optical imaging with otherpreoperative imaging modalities and topographical/depth information froma 3D scanning module. It also teaches that the depth information can beused to register images between multiple intraoperative optical imagingmodalities, such as NIR fluorescence, color RGB, or hyperspectral (usinga tunable liquid-crystal filter or a filter wheel), but not betweenindividual channels of a multi-channel imaging system. Whilehyperspectral imaging (or other modalities such as imaging polarimetry)can be a potentially valuable source of information in a digital loupesystem, the methods suggested in the prior art do not allow for theeffective combination of miniaturization, temporal resolution, spatialresolution, and light throughput that would be desired for an optimalsystem.

A third challenge that the present disclosure overcomes is related toergonomics. A traditional or analog surgical loupe comprises a pair ofnon-inverting telescopes that are suspended in front of a user's eyes,the optical axes of the left and right telescopes alignedcorrespondingly with the optical axes of the user's left and right eyes.There are three prototypical solutions for suspension of thesetelescopes, or oculars, in front of the user's eyes, within the priorart. Each has advantages and disadvantages with respect to functionalattributes of the loupe system, including weight, comfort, field ofview, view occlusion and peripheral vision, customization of fit,stability, and adjustability.

For the purposes of this disclosure, “weight” includes notions such asthe overall mass of the analog or digital surgical loupe or other visualaid, as well as the distribution of that mass on the head of thesurgeon. These both have implications for the comfort of the surgeon.For example, if the mass of such a system is distributed such that inoperation, it shifts the combined center of gravity of the system andthe surgeon's head significantly forward of the center of gravity of thesurgeon's head alone, this will increase strain on the surgeon's neckrelative to the unaided surgeon. Such strain contributes to thediscomfort of the surgeon especially in cases of prolonged use.Furthermore, distributing the weight of the system across a larger areaof the surgeon's head generally provides greater comfort thandistributing the weight across a smaller area of the surgeon's head,although certain areas of the head are more sensitive to pressure thanother areas, e.g., the temples and supraorbital regions being affectedby tight headbands, as well as the nose when used to support loupes vianose pads.

Field of view and view occlusion or peripheral vision are also importantfunctional attributes that are useful for comparing loupe systems. Here,by field of view we are referring to the apparent field of view, whichis the angular extent of the magnified field as presented to the user.This is to be distinguished from the true field of view, which is theangular extent of the unmagnified field. An eyepiece with a given clearaperture of the front lens surface supports a greater apparent field ofview when it is closer to the user's eye, or when the eye relief issmaller, than when it is further away. However, the closer the eyepieceis to the eye, the more the peripheral vision of the user is occluded.An ideal system would not occlude any parts of the user's field ofvision outside of the apparent field of the loupe system. In practicethis is not possible as the eyepiece must be stably mechanicallysupported and aligned with and in front of the optical axis of theuser's eye. Careful consideration of the support mechanisms, as in thepresent disclosure, can be used to minimize the perturbation of theuser's field of vision from these support mechanisms and thus preservethe sensation of an open view of the user's surroundings.

Finally, the interrelated attributes of customization of fit, stability,and adjustability are significant for determining the overallperformance of the loupe system. As a general rule, the more adjustablethe fit of a system is, the less it has to be customized to the user.However, to create a mechanism that is both adjustable and stablegenerally requires more material, and thus more mass, than a system thatis stable but not adjustable. This excess material has the potential toincrease the weight and view occlusion of the system, negativelyimpacting comfort and visual performance.

We now turn to a description of the design solutions presently in usefor analog surgical loupes. The first solution, which we shall call the“through-the-lens” mount, is the lowest profile, but also the leastflexible of the three. A pair of spectacles is custom-fit for thesurgeon through an involved fitting process. The working distance,interpupillary distance, declension angle, prescription, frame size, andprecise drilling positions must all be carefully measured andincorporated at the time of manufacture, and subsequently cannot bechanged. A hole is drilled into each of the left and right lenses of thespectacles, and these holes are used to support the oculars in aposition precisely aligned with the optical axes of the user's eyes in anear-vision position. This level of customization and lack ofadjustability is feasible because like eyeglasses, surgical loupes arenot traditionally shared. Also, custom loupes incorporate the surgeon'soptical prescription both within the spectacles and the telescopes, sothe peripheral field viewed through the spectacles remains in focus.This solution has the lowest weight as no framing is required beyond theeyeglasses. However, the bulk of the weight is supported by the nosepads resting on the surgeon's nose, thus this style becomesuncomfortable at higher magnifications due to the weight of the largeobjectives needing to be supported by these nose pads. Furthermore,placement of the loupes (and thus the maximum declension angle of theoculars) is somewhat constrained by the surgeon's anatomy, e.g., theheight of the surgeon's nose relative to the eyes. The through-the-lensplacement enables smaller oculars for the same apparent field of viewbecause the oculars can be placed very close to the surgeon's eyes. Butif changes in prescription are needed, the loupe system needs to beremanufactured. Furthermore, laser safety eyewear is not easilyintegrated with such a loupe.

A next style of loupe is the flip-up mount or front lens mount, which isclipped onto the front of the spectacles. The oculars are supportedcompletely in front of the spectacles via an adjustable support arm.This allows for more adjustment of lens position, declension, etc., andless need for customization. However, the weight of the system,supported primarily by the nose pads, increases significantly: biggerlenses are needed to maintain the same apparent field of view becausethe lenses now sit further away from the surgeon's eyes; more framing isneeded to support the lenses in an adjustable way; and finally, due tothe forward center of gravity relative to through-the-lens loupes, moreforce is placed on the surgeon's nose, and there is more strain on thesurgeon's neck. The presence of the support system in front of and abovethe surgeon's nose partially occludes the surgeon's field of vision nearthe center, and gives a somewhat uncomfortable experience relative tonot having anything there. Flexibility is enhanced as the spectaclelenses can be changed to enable changes in prescription or addition oflaser or other optical filters. While adjustment of ocular positioningis enabled by this mount, it is only possible over a relatively smallrange due to the need to keep the ocular support system small tominimize view occlusion, as well as due to the relatively short lengthof the ocular support arm. The adjustable declension is useful in thatit allows the surgeon to assume various cervical spine extension angleswhile viewing the same magnified work area, but as the lenses stick outmore than in the through-the-lens style of loupe, there is a greaterchance of interference with conventional face shields.

A third style of loupe is the flip up mount but with the support on aheadband rather than on the front of the spectacles. This relieves thenose from supporting the weight of the oculars and thus is suited tohigher magnifications and/or prismatic loupes that utilize a Keplerianrather than Galilean structure, with a prism to undo the image inversioncaused by the Keplerian telescope. A larger support structure/longersupport arm is needed to hold the oculars in front of the eyes of thesurgeon, necessitating even more weight, but this weight can bedistributed across the head using the headband structure. The longersupport arm may therefore appear even more prominently in the surgeon'speripheral vision, especially at greater ocular declension angles, anundesirable feature of this configuration. While a longer or largersupport structure generally enables longer translational ranges andgreater distances between pivot points and supported objects, thusenabling greater adjustment freedom, this comes at the expense ofstability, as rotational head motions are amplified by the longer leverarm. But personal eyewear, including laser safety eyewear, isindependent of the loupe system and therefore easily used in combinationwith it. Such a loupe system can be easily shared among surgeons.

Many of the considerations and tradeoffs that arise in the field ofsurgical loupes also arise in the field of near-eye displays orhead-mounted displays, especially those that provide for visualaugmentation. These include weight and comfort, stability andadjustability of fit, and preservation (or not) of peripheral vision. USPatent Publication US20040113867A1 teaches a head-mountable displaysystem that is designed to minimize the view occlusion of the user whilemaintaining the ability to see above and/or below the displays. The viewangle of the displays relative to the horizontal, commensurate with thedeclension angle of the loupes, is adjustable, as are various fittingparameters of the system, to enable a more comfortable fit and betterviewing experience in terms of reducing strain and preserving contextualawareness. U.S. Pat. No. 7,319,437 teaches a lightweight binocularnear-eye display that preserves the forward-looking peripheral vision ofthe user, though it does not specifically describe the mechanisms forhow to accomplish this in a way that could be flexible enough for alarge range of head sizes and shapes.

The telescopes of an analog surgical loupe are sometimes called oculars,though the words “ocular” and “eyepiece” can also be usedinterchangeably to describe the lens system or lens element closest tothe user's eye in an optical system designed for a human user. The word“objective” is often used to describe the front-most lens of a telescopefacing the object or work area. For an analog loupe, absent any foldingof the optical path using reflective or refractive means (which againadds bulk and weight), the optical axes of the objective and theeyepiece are collinear. As stated previously, an advantage of a digitalloupe is the bifurcation of the imaging side, comprising a stereo camerapair, and the display side, comprising a binocular near-eye display,into two distinct entities. Information transfers electronically betweenthem, and there is no requirement for their optical axes to be collinearor even aligned. This is advantageous because the means of support forboth entities can be optimized independently with respect to factors ofadjustability, stability, peripheral vision, and ergonomics. Forexample, by introducing parallax between or displacing the relativeviewpoints of the stereo camera pair and the user's eyes, it is possibleto have concurrent direct and augmented views of an object. Also,telescopes are generally understood as afocal optical systems (incomingand outgoing light beams are approximately collimated) that provideangular magnification. An angular shift of the telescope thereforecauses a magnified shift in the image viewed through the telescope.However, with bifurcated objective and eyepiece, we must consider howangular shifts of each of these subsystems affect the viewed image: anangular shift of the objective is magnified when viewed at the eyepiece,whereas an angular shift of the eyepiece is not magnified. Therefore,the stability requirements of the objective are greater than those ofthe eyepiece by the magnification factor.

Furthermore, the magnification factor of a telescope generally comesfrom the longer focal length of the objective relative to the eyepiece;therefore, the objective is correspondingly larger and heavier than theeyepiece. To minimize the forward pull of the center of gravity beyondthat of the surgeon's head alone, it is advantageous to mount the stereocamera pair (objective) of a digital loupe behind the displays(oculars/eyepieces), moving the center of gravity backward in a way thatis not possible with conventional analog loupes. Also, the onlyadjustment on the objective end that is needed is the declension angle,as opposed to the oculars/eyepieces, which need to be precisely alignedwith the optical axes of the user's eyes.

Accordingly, there is a need for a new kind of ocular support system,that could be used with analog loupes, digital loupes, head-mounteddisplays, or any head-worn optical system that includes an ocular, thatpreserves peripheral vision and thus preserves the user's contextualawareness and a sense of an open view, and that is lightweight, easilyadjustable, and stable. The present disclosure aims to provide such anocular support system that is especially suited to a digital loupesystem, where the supports for the oculars and the stereo camera paircan be separately optimized and adjusted, enabling concurrence of directand augmented vision.

While the devices and methods of the prior art lay a strong foundationfor a powerful visual aid for surgery, key gaps remain with regard tothe physical and visual ergonomics of such a system, specifically withregard to: minimizing double-vision with stable automatic convergence;preserving peripheral field and a comfortable concurrence of visionbetween the magnified or augmented view of an object and a direct viewto that object, in a form that is comfortable, stable, and easilyadjustable; and the incorporation of advanced optical imaging modalitiessuch as hyperspectral and multi-channel fluorescence imaging withoutcompromising image quality or spatial or temporal resolution. It is theaim of the present disclosure to fill these gaps as well as to provideseveral key enhancements that make the digital loupe an attractive andviable tool for augmenting a surgeon's vision.

SUMMARY OF THE DISCLOSURE

Aspects of the present disclosure provide a digital loupe that combinesfreedom of movement and flexible working distance, ergonomic comfort,open peripheral vision, concurrence between magnified (or augmented)vision and normal unobstructed vision, magnification with high imagequality, and optionally, advanced optical imaging modalities to augmenta surgeon's vision in real time. These aspects achieve such advantagesvia a specific means of supporting oculars in front of the eyes of thesurgeon in addition to a specific arrangement of distance sensing,camera calibration, and image transformations that present astereoscopic augmented view of a surgical wound to a surgeon in anoptimal way. Unlike with a surgical microscope, the freedom of movementand flexible working distance enabled by aspects of the presentdisclosure allow the surgeon to quickly and naturally integrate views ofthe surgical wound from multiple viewpoints. And unlike with traditionalanalog loupes, the open peripheral view and concurrence of direct andaugmented views allow the surgeon to maintain maximal contextualawareness of the surgical operation, ensuring a smoother outcome.

In one embodiment, the digital loupe comprises a stereo camera pairmounted to the head of a user, including a depth sensing element thathas a sensing direction which nominally bisects the lines of sight oroptical axes of the cameras of the stereo pair. The depth sensingelement may give a single non-spatially-resolved measurement or aspatially-resolved measurement. It may have a defined field of view thatmay depend on a magnification of the digital loupe. The digital loupemay include illumination also nominally directed along a line thatbisects the lines of sight of the cameras, parameters of which mayadjust in response to the distance to the subject or object underobservation. It may also include a binocular head-mounted display, and aprocessor that is in operative communication with the stereo camerapair, the depth sensing element, and the binocular head-mounted display.The processor may be in operative communication with an illuminationcontroller that controls an illumination source to adjust parameters ofthe illumination, such as the illumination intensity and spatialdistribution or extent of intensity, as a function of distance measuredby the distance sensor. The illumination may be pulsed, potentially in amanner synchronized with the exposure intervals of the stereo camerapair.

The lines of sight of the stereo camera pair may intersect at a nominalworking distance of a user, which could be, for example, the averagedistance between the eyes and hands of a surgeon in an operatingposture, or the average of such distances across a set of surgeons. Thedifference between the system's predefined nominal working distance andthe actual working distance between a user's eyes and hands should besmall. Furthermore, the eyepieces of the binocular head-mounted displaymay have a similar convergence angle such that the optical axes of theleft and right displays intersect at a similar nominal working distance.The head-mounted display may have a virtual image distance, or distancebetween the user's eyes and the virtual image plane formed by thehead-mounted display optics, similar to a nominal working distance, andit may be designed so as to preserve the peripheral or “far” vision ofthe user. For example, the oculars of the head-mounted display can beplaced in a near-vision position familiar to users of traditionalthrough-the-lens loupes, with ocular supports that only minimallyobscure peripheral vision. This allows the user to switch back and forthbetween normal vision, or direct vision of the surgical wound above orbelow the oculars, and magnified or augmented vision through the ocularsof the head-mounted display, with only an eye rotation (i.e., with nohead movement) and with minimal change in visual accommodation andvergence, thus maximizing visual and ergonomic comfort and reducingeyestrain. The direct view and augmented view are therefore“concurrent.” In order to further accommodate seamless transitionsbetween direct and augmented vision, the digital loupe system can varyone or more of a virtual convergence angle of images within the oculars,a real convergence angle of the oculars themselves, and a virtual imagedistance, in response to information derived from the distance sensor,preferably to minimize changes in visual accommodation and vergence whenswitching between direct and augmented views of an object.

The processor can be used to store and update calibration informationthat models the precise alignment of the cameras of the stereo pair(e.g., intrinsic and extrinsic camera matrices as used in the pinholecamera model) or other subsystems, including relative position andorientation of all cameras, distance or other sensors, sources ofillumination, and displays or oculars. Depending on ergonomic ormechanical degrees of freedom and relative placement of these differentsubsystems in the digital loupe, it may be necessary to track the stateof these degrees of freedom in order to have a complete picture of therelative position and orientation of each of these subsystems. However,such a complete picture is only needed for some embodiments of thepresent disclosure.

Minimally it is important to calibrate the cameras of the stereo camerapair, as there will always be slight differences between cameraparameters (position, orientation, sensor position relative to lensoptical axis, focal length, and pixel size) as designed, and as realizedin practice. These slight differences, in addition to the convergenceangle of the stereo camera pair as designed, manifest in image shiftsthat vary with distance to the subject under observation, which may,especially at high magnifications, sufficiently displace left and rightimages of the stereo camera pair such that they cannot be vieweddirectly through a binocular head-mounted display without furthertransformation. With knowledge of the camera calibration information,combined with knowledge of the distance to the subject from a distancesensor, it is possible for a processor to precisely correct for theeffects of slight camera misalignments as a function of distance. Theprocessor can translate or transform the images before displaying themsuch that they appear to have come from a stereo camera pair withoptical axes that converge to a point along the optical axis of thedistance sensor and at the distance measured by the distance sensor.That is, it appears to the user as if the cameras were both directedprecisely toward the subject, directly in front of the stereo camerapair at a distance given by the distance sensor. Because the images aresubsequently viewed by the user in the head-mounted display, withoptical axes of the left and right eyepieces converging to a nominalworking distance, the magnified view of the subject will appear at thecenter of each display, and thus also at the nominal working distance.Thus, because the nominal working distance is the same as, or close to,the actual working distance between the user's eyes and the subject, auser can switch between looking at the subject directly and looking atthe subject through the eyepieces with minimal change in the vergencestate of their eyes. The processor can optionally perform a secondtransformation of the images before display, based on the measureddistance, such that the displayed subject appears at the actual, ratherthan nominal, working distance. This second transformation would beequivalent to virtually adjusting the relative convergence angle of thetwo oculars, such that the left and right eyes converge at the actualworking distance (e.g., as measured by the distance sensor) when viewingthe left and right images of the subject with single vision.Furthermore, if variable focus oculars are used, the processor canmodify the virtual image distance to match the actual working distance.Thus, in this optional approach, no change in visual accommodation orvergence would be necessary to switch between a magnified or augmentedview of the subject and a direct view of the subject above or below theoculars.

If an imaging distance or depth sensor is used, or the geometry of thescene is estimated (for example by disparity calculations from thestereo pair, perhaps made more accurate with the point depth sensor, orvia structure from motion algorithms), it would be possible to fullyadjust scene parallax. One example scenario where this ability would beuseful is to transform the viewpoints of the cameras of the stereo pairto match the viewpoints of the user's eyes. It is advantageous to mountthe cameras as close to the head as possible to minimize the lever armwith respect to the head, as this makes the most stable image; apreferred mounting position is therefore on the crown of the user'shead. The vertical displacement of the stereo camera pair relative tothe user's eyes introduces vertical parallax to the viewed image thatcan be mitigated via the appropriate transformation. While spatiallyresolved depth information would enable a full correction of sceneparallax, it is also possible to correct for the average scene parallaxwith only a point distance sensor. If the relative geometry of theeyepieces and the stereo camera pair is known, then the average sceneparallax can be adjusted as a function of measured distance, bytransforming or shifting the image of the subject such that it appearsas if the stereo camera pair were always directed toward the subject.

Additional cameras can be used to include other modalities, such asfluorescence imaging, polarization imaging, hyperspectral imaging, etc.With an imaging depth sensor, it is possible to mount, for example, anNIR fluorescence imager, a polarization imager, and a color RGB stereopair side by side, and use the spatially resolved depth information tocorrect for parallax and map fluorescence or other information onto theviewpoints of the stereo camera pair, or onto the viewpoints of theuser's eyes. The processor can include extrinsic and intrinsic cameracalibration matrices or other camera models in order to properly mapbetween the different viewpoints with minimal registration error andwithout requiring computationally costly and error-prone iterativeregistration algorithms.

It is an aspect of the present disclosure to provide a novel form ofmulti-channel imager that is more amenable to a digital loupe than thoseof the prior art. Here, multi-channel imager refers to imagingmodalities that are traditionally thought of as using a single device,such as a hyperspectral imager or imaging polarimeter, that outputs an“image cube”, which is a stack of individual 2D images corresponding tosingle channels such as wavelength bands or polarization components,etc. Such imaging technologies may be large and bulky and thus notamenable to integration in a digital loupe; also, depending on thetechnology, they may not have adequate spatial and/or temporalresolution or light throughput. Rather, by using a calibrated array ofminiature cameras, each one recording one slice or channel of an imagecube corresponding to a given modality, one can use information from animaging depth sensor to remove parallax from each camera of the arrayand synthesize a full image cube as if it were recorded simultaneouslyfrom a single viewpoint. This technique of the present disclosure has anadvantage over sensors that tile various spectral or polarizationfilters at the pixel level as it preserves spatial resolution and allowsfor flexibility of integration, choice of filters, and independentsensor and exposure parameters. Also, it has an advantage overtemporally scanning sensors as there is no temporal scanning involved.Thus, the multi-channel imaging technique of the present disclosureenables real-time integration of images from multiple disparate opticalimaging modalities within a digital loupe system.

The present disclosure is also directed toward ocular support structuresespecially suited for use in digital loupe systems. Throughout thisdisclosure, the word “ocular” can be used to describe any opticalelement or system of elements mounted in front of the eye for purposesof viewing or visualization by the eye, such as a telescope in the caseof analog loupes, or an eyepiece, with or without an adjacentmicrodisplay, in the case of a head-mounted display or near-eye display.Many embodiments of the disclosure concern novel means of supporting andaligning an ocular in front of the eye of a user while improving uponthe prior art in terms of better overall ergonomics, includingperipheral vision, comfort, stability and adjustability, etc. Oneembodiment of this disclosure may occur within the context of a digitalloupe system, such that the visual output of such a system can bedisplayed in a manner that allows for comfortable, stable use over themultiple hours of a surgical operation, allowing the surgeon to selectthe most ergonomic operating positions while minimizing the occlusion tothe surgeon's peripheral vision.

Embodiments of this disclosure comprise judicious placement of thesupport arm or support arms of an ocular with respect to the anatomy ofa human head. In some embodiments, ocular support arms or systemsdescribed herein do not include the lens barrel or immediate enclosureof a lens or ocular. Rather, they comprise the linkage that mechanicallyconnects the ocular to the user, or any number of mechanical linkagesaway from the ocular, starting with the most adjacent one. Embodimentsof the present disclosure may comprise ocular support arms, structures,or systems, that keep weight off the nose and other sensitive parts ofthe head and face while maintaining as much peripheral vision, or asmuch of an open view, as possible. Some embodiments are directed towardocular support systems comprising multiple articulation points thatenable full positioning adjustment of the oculars, or components of suchsystems, such as headbands, that better enable such systems to performas desired. Other embodiments are directed toward placement of ocularsupport arms with respect to the wearer's head, or with respect to theuser's field of vision. Further embodiments take into account relativeplacement of the stereo camera pair, such that the declension of thestereo camera pair can be separately adjusted from that of the ocular,enabling both a more vertical operating posture as well as concurrenceof a view of a subject through the ocular as captured by the stereocamera pair and a view of the same subject above or below the ocular.

A digital loupe system is provided, comprising a stereo camera pairadapted and configured to generate image signals of an object or workarea, a distance sensor adapted and configured to obtain a measurementof distance to the object or work area; and a processor operablyconnected to the stereo camera pair and the distance sensor, wherein theprocessor comprises a memory configured to store camera calibrationinformation relating to the stereo camera pair and to perform atransformation to image signals from the stereo camera pair based on adistance measurement from the distance sensor and the camera calibrationinformation.

In some implementations, the transformation causes the image signal toappear as if generated from a stereo camera pair with optical axes thatconverge at a distance corresponding to the distance measurement.

In other implementations, the distance sensor has a field of view thatis adjustable. In some examples, the field of view of the distancesensor is adjustable based on a magnification of the digital loupesystem. In other implementations, the optical axis of the distancesensor approximately bisects the angle formed by the optical axes of thestereo camera pair. In another implementation, the distance sensor is animaging distance sensor. In another implementation, the distance sensorhas a narrow, collimated beam.

In one embodiment, the stereo camera pair is adapted to be mounted onthe crown or forehead of a user's head.

In some implementations, a declination angle of the stereo camera pairis adjustable.

In other implementations, each camera of the camera pair has an opticalaxis, the optical axes of the stereo camera pair being configured toconverge at a distance approximately equal to an intended workingdistance of a user.

In some examples, the digital loupe system further comprises a binocularhead-mounted display comprising first and second displays operablyconnected to the processor to receive the image signals from theprocessor generated by the stereo camera pair and to display images fromthe image signals. In some examples, the transformation causes theimages to appear as if the stereo camera pair had optical axes thatconverge at a distance corresponding to the distance measurement. Inother implementations, the head-mounted display is configured to have avirtual image distance corresponding approximately to a working distanceof a user. In some implementations, the displays are mounted in anear-vision position. In another implementation, the processor isfurther configured to display the image signals in the displays with aspatially-varying magnification. the binocular head-mounted display canfurther comprise an ambient light sensor, the processor being furtherconfigured to use a signal from the ambient light sensor to adjust adisplay characteristic of the head-mounted display. In some examples,the optical axes of the head-mounted display converge at a distanceapproximately equal to a working distance of a user.

In some implementations, the processor of the digital loupe system isfurther configured to use distance information from the distance sensorto shift a viewpoint of the image signals.

In another implementation, the stereo camera pair comprises a colorcamera that provides color image signals to the processor. In someimplementations, the processor is further configured to process thecolor image signals using a 3-dimensional look-up table. In otherexamples, the processor is further configured to process the color imagesignals to substitute colors from a region in a color space where a useris less sensitive to changes in color to a second region in the colorspace where a user is more sensitive to changes in color.

In some embodiments, the system is configured to perform imagestabilization through optical image stabilization at the stereo camerapair or through electronic image stabilization at the processor.

In other embodiments, the cameras are configured to automaticallymaintain focus.

In one implementation, the system further comprises a source ofillumination adapted to illuminate the object or work area. In someexamples, the source of illumination is controlled by an illuminationcontroller that adjusts a parameter of the illumination based uponmeasurements of distance from the distance sensor. In other examples,the illumination may be pulsed in a manner synchronized with an exposureinterval of the stereo camera pair.

In some examples, at least one image sensor in the stereo camera pair isan RGB-IR sensor. In another implementation, the at least one imagesensor has a high dynamic range capability.

In some examples, the system further comprises an additional imagingmodality different from the one that the stereo pair comprises. Forexample, the additional imaging modality can comprise a multi-channelimaging system.

A multi-channel imaging system is further provided, comprising an arrayof at least two cameras, wherein at least two channels are distributedacross the at least two cameras, an imaging distance sensor adapted andconfigured to image a field of view similar to a field of view imaged bythe at least two cameras, and a processor configured to store cameracalibration information regarding the at least two cameras, wherein thecamera calibration information is defined in a coordinate systemrelative to the imaging distance sensor, wherein the processor isconfigured to receive image signals from the at least two cameras anddepth information from the imaging distance sensor and to use the depthinformation and the camera calibration information to correct forparallax between the at least two cameras, thus providing amulti-channel image that appears to originate from a single viewpoint.

In some implementations, the system is a multispectral imaging system,the channels correspond to spectral bands, and the multi-channel imagecomprises a hyperspectral image.

In another implementation, the system is an imaging polarimeter, thechannels correspond to polarization combinations, and the multi-channelimage comprises a polarimetry image.

A method of obtaining a stereoscopic image of an object is alsoprovided, the method comprising obtaining first and second images of anobject with first and second cameras, obtaining a measurement ofdistance to the object with a distance sensor, and applying atransformation to the first and second images using the measurement ofdistance and using calibration information of the first and secondcameras.

In some examples, the method further comprises displaying thetransformed first and second images on first and second displays,respectively. Additionally, the method can comprise supporting the firstand second displays in a field of vision of a user. In someimplementations, optical axes of the first and second displays convergeat a distance approximately equal to a working distance between the userand the object. In one example, the step of applying the transformationcomprises virtually adjusting the convergence angle of the first andsecond displays. In another example, the step of applying atransformation comprises causing the first and second images to appearon the first and second displays as if the first and second cameras hadoptical axes that converge at a distance corresponding to themeasurement of distance.

In some embodiments, the applying step comprises adjusting a field ofview of the first and second images using the measurement of distance.

In other embodiment, the method further comprises using the measurementof distance to shift a viewpoint of the first and second images.

In some implementations, the method further comprises changing amagnification of the first and second images and adjusting a field ofview of the distance sensor with the change of magnification.

In another embodiment, the method further comprises changing thedistance between the object and the first and second cameras andadjusting the transformation with the change in distance.

The method can additionally include illuminating the object. In someexamples, the illuminating step comprises determining an illuminationparameter based upon the measurement of distance and illuminating theobject based on the illumination parameter. In another example, theilluminating step comprises pulsing an illumination source in a mannersynchronized with exposure intervals of the first and second cameras.

A method of viewing an object is also provided, comprising engaging ahead engagement member with a user's head, the head engagement membersupporting two cameras above the user's head, placing each of a firstdisplay and a second display in a line of sight with an eye of the user,obtaining first and second images of the object with first and secondcameras, obtaining a measurement of distance to the object with adistance sensor supported by the head engagement member, applying atransformation to the first and second images using the measurement ofdistance and using calibration information of the first and secondcameras, and displaying the transformed first and second images on firstand second displays, respectively.

In some implementations, the method further comprises supporting thefirst and second displays with the head engagement member.

In one example, optical axes of the first and second displays convergeat a distance approximately equal to a working distance between the userand the object.

In one implementation, the step of applying the transformation comprisesvirtually adjusting the convergence angle of the first and seconddisplays.

In another example, the step of applying a transformation comprisescausing the first and second images to appear on the first and seconddisplays as if the first and second cameras had optical axes thatconverge at a distance corresponding to the measurement of distance.

In some embodiments, the applying step comprises adjusting a field ofview of the first and second images using the measurement of distance.

In other embodiments, the method further comprises using the measurementof distance to shift a viewpoint of the first and second images.

In another embodiment, the method further comprises changing amagnification of the first and second images and adjusting a field ofview of the distance sensor with the change of magnification.

In some examples, the method further comprises changing the distancebetween the object and the first and second cameras and adjusting thetransformation with the change in distance.

In one embodiment, the method further comprises illuminating the objectwith an illumination source supported by the head engagement member. Insome examples, the illuminating step comprises determining anillumination parameter based upon the measurement of distance andilluminating the object based on the illumination parameter. In otherexamples, the illuminating step comprises pulsing an illumination sourcein a manner synchronized with an exposure interval of the first andsecond cameras.

A method of obtaining a multi-channel image is also provided, the methodcomprising obtaining at least first and second images of an object fromat least first and second cameras, obtaining a depth image of an objectusing an imaging depth sensor, and applying a transformation to the atleast first and second images based on the depth image and calibrationinformation of the at least first and second cameras, wherein the atleast first and second images correspond to single channels of amulti-channel imaging modality, and the transformation removes parallaxbetween the at least first and second images.

In some examples, the channels correspond to spectral bands, and themulti-channel image comprises a multispectral image. In other examples,the channels correspond to polarization combinations, and themulti-channel image comprises a polarimetry image.

A head mounting system for supporting a pair of oculars within a line ofsight of a human user is also provided, the head mounting system beingadapted to be worn by the user, the system comprising a head engagementmember adapted to engage the user's head, and first and second supportarms each having a proximal portion supported by the head engagementmember, a distal portion disposed so as to support an ocular in theuser's line of sight, and a central portion disposed between theproximal portion and the distal portion, the head mounting system beingconfigured such that when the head engagement member is engaged with theuser's head, the central portion of each support arm is configured toextend laterally and superiorly from the distal portion toward theproximal portion without extending through a region of the user's facemedial and superior to the user's eyes and inferior to the user'sglabella, and the proximal portion of each support arm is arranged andconfigured to be disposed medial to the central portion.

In some implementations, the proximal portion of each support arm isfurther configured to be disposed medial to the user's frontotemporaleswhen the head engagement member is engaged with the user's head.

In one embodiment, the central portion of each support arm is furtherconfigured to extend posteriorly from the distal portion toward theproximal portion without extending through a region of the user's facemedial and superior to the user's eyes and inferior to the user'sglabella when the head engagement member is engaged with the user'shead.

In some examples, the proximal portions of the first and second supportarms are each connected to the head engagement member by a hinge adaptedto allow an angle between the support arms and the head engagementmember to be changed. In one implementation, the hinge is adapted toallow the proximal, central, and distal portions of the support arms tobe moved above the user's eyes when the head engagement member isengaged with the user's head.

In some examples, the first and second support arms are each supportedby a sliding connector allowing a height of the support arms withrespect to the head engagement member to be changed.

In another embodiment, each of the first and second support armscomprises multiple segments. In one embodiment, the system furthercomprises a connector connecting adjacent segments of each support arm.In some implementations, the connector is adapted and configured toallow an effective length of a segment of the support arm to beadjusted.

In one example, the distal portion of each of the first and secondsupport arms comprises a display bar adapted to be connected to anocular of the pair of oculars. In one embodiment, the first support armdisplay bar is integral with the second support arm display bar. Inanother embodiment, the first support arm display bar and the secondsupport arm display bar are not connected. In another embodiment, thesystem further comprises first and second hinges connecting the displaybar to the central portions of the first and second support arms,respectively. In one example, the hinges are adapted and configured toallow a declension angle of oculars attached to the display bar to bechanged. In another example, the hinges are adapted and configured toallow the first and second support arms to be moved toward or away fromthe user's head.

In some embodiments, the head engagement member comprises a headband. Insome examples, the headband is adjustable to fit different user headsizes.

In one embodiment, the head engagement member comprises a plurality ofpieces adapted to engage the user's head, the plurality of pieces beingconnected by a flexible connector.

In another embodiment, the head engagement member comprises a connectoradapted to connect to a head strap.

In some embodiments, the first and second support arms are two ends of aunitary support arm. In one example, the unitary support arm has a ram'shorn shape. In another example, the unitary support arm has a partialrectangle shape.

In some embodiments, the system further comprises a transparent windowattached to the ocular supports and adapted to protect the user's face.

In other embodiments, the system comprises a sensor configured to reporta state of an articulation of the head mounting system.

In one example, an articulation of the head mounting system is adaptedto be automatically actuated.

In one implementation, the system further comprises a linkage betweenthe first and second support arms, the linkage being configured toactuate a portion of one of the support arms in response to an actuationof a corresponding portion of the other support arm.

An imaging system adapted to be worn by a human user to provide a viewof a work area is further provided, the system comprising a headmounting subsystem for supporting a pair of oculars within a line ofsight of a human user, the head mounting system being adapted to be wornby the user, the head mounting subsystem comprising a head engagementmember adapted to engage the user's head, and first and second supportarms each having a proximal portion supported by the head engagementmember, a distal portion disposed so as to support an ocular in theuser's line of sight, and a central portion disposed between theproximal portion and the distal portion, the head mounting system beingconfigured such that when the head engagement member is engaged with theuser's head, the central portion of each support arm is configured toextend laterally and superiorly from the distal portion toward theproximal portion without extending through a region of the user's facemedial and superior to the user's eyes and inferior to the user'sglabella, and the proximal portion of each support arm is arranged andconfigured to be disposed medial to the central portion, two camerassupported by the head engagement member, first and second ocularssupported by the distal portions of the first and second support arms,respectively, so as to be positionable in the user's line of sight whenthe head engagement member is engaged with the user's head, and aprocessor adapted and configured to display in displays of the ocularsimages obtained by the two cameras.

In some embodiments, the proximal portion of each support arm is furtherconfigured to be disposed medial to the user's frontotemporales when thehead engagement member is engaged with the user's head.

In one embodiment, the central portion of each support arm is furtherconfigured to extend posteriorly from the distal portion toward theproximal portion without extending through a region of the user's facemedial and superior to the user's eyes and inferior to the user'sglabella when the head engagement member is engaged with the user'shead.

In another embodiment, the proximal portions of the first and secondsupport arms are each connected to the head engagement member by a hingeadapted to allow an angle between the support arms and the headengagement member to be changed. In some examples, the hinge is adaptedto allow the proximal, central, and distal portions of the support armsto be moved above the user's eyes when the head engagement member isengaged with the user's head.

In one embodiment, the first and second support arms are each supportedby a sliding connector allowing a height of the support arms withrespect to the head engagement member to be changed.

In some implementations, each of the first and second support armscomprises multiple segments. In one example, the system furthercomprises a connector connecting adjacent segments of each support arm.In other embodiments, the connector is adapted and configured to allowan effective length of a segment of the support arm to be adjusted.

In one implementation, the system further comprises first and secondocular supports adapted to change a distance between the oculars.

In some examples, the head mounting subsystem is configured to permit adeclension angle of the oculars with respect to the user's line of sightto be changed.

In another implementation, the distal portion of each of the first andsecond support arms comprises a display bar supporting the first andsecond oculars. In one example, the first support arm display bar isintegral with the second support arm display bar. In another example,the first support arm display bar and the second support arm display barare not connected. In one embodiment, the system further comprises firstand second hinges connecting the display bar to the central portions ofthe first and second support arms, respectively. In one embodiment, thehinges are adapted and configured to allow a declension angle of theoculars to be changed. In another embodiment, the hinges are adapted andconfigured to allow the first and second arms to be moved toward or awayfrom the user's head.

In some examples, the head engagement member comprises a plurality ofpieces adapted to engage the user's head, the plurality of pieces beingconnected by a flexible connector.

In other examples, the first and second support arms are two ends of aunitary support arm.

In some embodiments, each of the first and second support arms has aram's horn shape.

In another embodiment, each of the first and second support arms has apartial rectangle shape.

In one embodiment, the system further comprises a transparent windowattached to the ocular supports and adapted to protect the user's face.

In another example, the system further includes a distance sensorsupported by the head engagement member.

The system can comprise a camera mount movable with respect to the headengagement member to change a view angle of one or both of the cameras.

In one implementation, the system further comprises a transparent windowextending in front of the displays and adapted to protect the user'sface.

In some embodiments, the system further includes a source ofillumination supported by the head engagement member.

In another implementation, the system includes a sensor configured toreport a state of an articulation of the head mounting system.

In some implementations, an articulation of the head mounting system isadapted to be automatically actuated.

In another example, the system includes a linkage between the first andsecond support arms, the linkage being configured to actuate a portionof one of the support arms in response to an actuation of acorresponding portion of the other support arm. In one example, thelinkage comprises a sensor configured to sense an actuation state of theportion of one of the support arms and report the actuation state to theprocessor and an actuator configured to actuate the correspondingportion of the other support arm and to receive commands generated bythe processor, the processor configured to generate commands to theactuator in response to a report received from the sensor.

In one embodiment, the head engagement member comprises a headband. Insome examples, the headband is adjustable to fit different user headsizes.

In another embodiment, the head engagement member comprises a connectoradapted to connect to a head strap.

A method of viewing a work area is also provided, comprising engaging ahead engagement member with a user's head, the head engagement membersupporting two cameras above the user's head, placing each of twooculars in a line of sight with an eye of the user, the first and secondoculars supported by first and second supports arms, respectively,positioned such that a central portion of each support arm extendslaterally and superiorly from the oculars toward the head engagementmember without extending through a region of the user's face medial andsuperior to the user's eyes and inferior to the user's glabella,supporting each of the first and second support arms at a position ofthe head engagement member medial to the central portion of the firstand second support arms, respectively; and displaying in the ocularsimages of the work area obtained by the cameras.

In some examples, the supporting step comprises supporting each of thefirst and second support arms at a position of the head engagementmember medial to the user's frontotemporales.

In one embodiment, the central portion of each support arm also extendsposteriorly from the distal from the oculars toward the head engagementmember without extending through a region of the user's face medial andsuperior to the user's eyes and inferior to the user's glabella when thehead engagement member is engaged with the user's head.

In some examples, the method further includes viewing the work areaalong a line of sight extending over the oculars.

In another implementation, the method further includes viewing the workarea along a line of sight extending under the oculars.

In one embodiment, the method further includes viewing the work areasimultaneously through the oculars and around the oculars.

In another example, the method includes moving the oculars upward withrespect to the user's eyes.

In some implementations, the method comprises moving the ocularsdownward with respect to the user's eyes.

In another example, the method comprises changing a distance between theoculars.

In some embodiments, the method further includes adjusting a shape ofthe head engagement member to fit the user's head.

In some examples, the method includes moving at least one of the firstsupport arm and the second support arm medially or laterally.

In another example, the method includes moving the first and secondsupport arms above the user's eyes.

In some implementations, the method further comprises obtaining ameasurement of distance from the cameras to the work area and applying atransformation to images obtained by the cameras to create transformedimages, the displaying step comprising displaying the transformed imageson the oculars. In one example, the step of obtaining a measurement ofdistance from the cameras to the work area is performed by using adistance sensor supported by the head engagement member. In anotherexample, the step of applying the transformation comprises virtuallyadjusting the convergence angle of the first and second oculars. In oneimplementation, the step of applying a transformation comprises causingthe first and second images to appear on the first and second oculars asif the first and second cameras had optical axes that converge at adistance corresponding to the measurement of distance.

In one example, the method further comprises illuminating the object. Inone example, the illuminating step comprises determining an illuminationparameter based upon the measurement of distance and illuminating theobject based on the illumination parameter. In another example, theilluminating step comprises pulsing an illumination source in a mannersynchronized with exposure intervals of the first and second cameras.

In another embodiment, the method further comprises moving at least oneof the first and second support arms automatically.

In some embodiments, the method includes automatically moving at leastpart of the second support arm in response to movement of acorresponding part of the first support arm.

In one example, the method includes sensing an actuation state of one ofthe support arms.

A head mounting system for supporting an ocular within a line of sightof a human user is provided, the head mounting system being adapted tobe worn by the user, the system comprising head engagement memberadapted to engage the user's head, and a support arm having a proximalportion supported by the head engagement member, a distal portiondisposed so as to support an ocular in the user's line of sight, and acentral portion disposed between the proximal portion and the distalportion, the head mounting system being configured such that when thehead engagement member is engaged with the user's head, the centralportion of the support arm is configured to extend laterally andsuperiorly from the distal portion toward the proximal portion withoutextending through a region of the user's face medial and superior to theuser's eyes and inferior to the user's glabella, and the proximalportion of the support arm is arranged and configured to be disposedmedial to the central portion.

In some embodiments, the proximal portion of the support arm is furtherconfigured to be disposed medial to the user's frontotemporales when thehead engagement member is engaged with the user's head.

In another embodiment, the central portion of the support arm is furtherconfigured to extend posteriorly from the distal portion toward theproximal portion without extending through a region of the user's facemedial and superior to the user's eyes and inferior to the user'sglabella when the head engagement member is engaged with the user'shead.

In some examples, the proximal portion of the support arm is connectedto the head engagement member by a hinge adapted to allow an anglebetween the support arm and the head engagement member to be changed. Inone embodiment, the hinge is adapted to allow the proximal, central, anddistal portions of the support arms to be moved above the user's eyeswhen the head engagement member is engaged with the user's head.

In some implementations, the support arm is supported by a slidingconnector allowing a height of the support arm with respect to the headengagement member to be changed.

In another example, the support arm comprises multiple segments. In someexamples, the system further comprises a connector connecting adjacentsegments of the support arm. In one example, the connector is adaptedand configured to allow an effective length of a segment of the supportarm to be adjusted.

In some embodiments, the distal portion of the support arm comprises adisplay bar adapted to be connected to an ocular of the pair of oculars.In one example, the system further comprises a hinge connecting thedisplay bar to the central portion of the support arm.

In some embodiments, the hinge is adapted and configured to allow adeclension angle of an ocular attached to the display bar to be changed.In one example, the hinge is adapted and configured to allow the supportarm to be moved toward or away from the user's head.

In one embodiment, the head engagement member comprises a headband. Insome examples, the headband is adjustable to fit different user headsizes.

In another implementation, the head engagement member comprises aplurality of pieces adapted to engage the user's head, the plurality ofpieces being connected by a flexible connector.

In some examples, the head engagement member comprises a connectoradapted to connect to a head strap.

In another embodiment, the support arm has a ram's horn shape. Inanother example, the support arm has a partial rectangle shape.

In some implementations, the system includes a transparent windowattached to the ocular support and adapted to protect the user's face.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the disclosure are set forth with particularity inthe claims that follow. A better understanding of the features andadvantages of the present disclosure will be obtained by reference tothe following detailed description that sets forth illustrativeembodiments, in which the principles of the disclosure are utilized, andthe accompanying drawings of which:

FIG. 1 shows a surgeon operating with an example embodiment of thepresent disclosure.

FIG. 2 shows a schematic diagram of an embodiment of the presentdisclosure.

FIG. 3 shows a schematic diagram of an example binocular head-mounteddisplay of the present disclosure, including a working distance and aconvergence angle that are associated with a virtual image plane.

FIG. 4 is a schematic of a pair of cameras along with a distance sensorwhose optical axis nominally bisects the optical axes of the pair ofcameras.

FIG. 5 depicts a front projection of a head delineating preferredregions for routing an ocular support arm.

FIG. 6 shows a plot of the visual field of a user's left eye delineatingpreferred regions for routing an ocular support arm.

FIG. 7A is a perspective view of a digital loupe system.

FIG. 7B is a side view of a digital loupe system.

FIG. 7C is a front view of a digital loupe system.

FIGS. 8A-8C show different articulation states of a digital loupesystem.

FIGS. 9A-9D show further different articulation states of a digitalloupe system.

FIGS. 10A-10B show a segmented headband intended for use in a digitalloupe system.

FIGS. 11A-11D depict different views and articulation states of anocular support structure.

FIGS. 12A-12D depict different views and articulation states of anotherocular support structure.

FIGS. 13A-13D depict different views and articulation states of yetanother ocular support structure.

FIG. 14 depicts coupled side arms of an ocular support structure.

FIG. 15 depicts part of an ocular support structure with oculardeclension coupled through the top member.

FIGS. 16A-B illustrate a face shield that can be used with the digitalloupe system of this invention.

DETAILED DESCRIPTION

FIG. 1 depicts a surgeon 100 operating on a wound 110 (i.e., a targettissue site or a surgical work area) and wearing an example embodimentof the present disclosure, comprising a sensing/illumination unit 120and a binocular head-mounted display (HMD) 130. The sensing unit 120 andHMD 130 are both operably connected to a processor, not shown. Thesensing unit 120 comprises a stereo camera pair that receives a stereoimage of the wound 110 and transmits the stereo image to the HMD 130.The HMD 130 has eyepieces or oculars 131 a,b that are mounted in a“near” vision position familiar to those who wear traditional surgicalloupes and also bifocals, in order to preserve “normal” or “far” vision.The surgeon 100 can either look directly at the wound 110, e.g., in the“far” vision position above the eyepieces of the HMD 130, or through theHMD 130 to see a magnified version of the wound 110. The virtual imagedistance of the HMD 130 is approximately the same as the workingdistance from the surgeon's eyes to the wound 110. Also, the opticalaxes of the HMD 130 converge to a nominal position of the surgical wound110 relative to the surgeon 100. Therefore, when the surgeon 100switches between looking directly at the wound 110 or through the HMD130, there is minimal change in the accommodation or convergence of hereyesight. As will be explained further below with regard to systemergonomics, the sensing unit 120 is mounted on top of the surgeon's headin order to have a stable mounting platform, as the potentially highmagnifications enabled by this system benefit from a stable mountingplatform for the cameras. Also, the displacement of the sensing unit 120with respect to the HMD 130, in a direction transverse to the opticalaxes of the HMD 130, is what enables the simultaneous and concurrentpresence of the direct and magnified views of the surgical wound 110 inthe surgeon 100's field of vision. The surgeon 100 can switch betweencentering the direct view or the magnified view of the wound 110 in thecenter of her field of vision with only an eye rotation and without theneed to move her head. The direct view around the HMD and the augmentedview in the HMD are therefore “concurrent.” The placement and support ofthe oculars of the HMD 130 is such that an open view of the surgicalwound 110 as well as the surgeon 100's surroundings is maintained formaximum contextual awareness during the surgical operation.

Note that as used herein, a stereo camera pair may comprise anyelectronic imaging device that outputs a signal that can be viewedstereoscopically with a suitable display. For example, it could comprisetwo color RGB cameras with a baseline separation, similar to theseparation of two eyes on a person, that afford for slightly differentviewpoints, thus providing a stereoscopic view when rendered on abinocular head-mounted display. Alternatively, it could comprise twoinfrared cameras, or other types of cameras or focal plane arrays. Asanother alternative, it could comprise a single plenoptic (lightfield)camera, where signals for left and right displays are virtually renderedby calculating the images derived from a shift in viewpoint. As yetanother alternative, it could comprise a single camera and a depthimager, where the information combined from single camera and depthimager is used to simulate a second viewpoint for stereopsis.

FIG. 2 shows a schematic diagram 200 of an embodiment of the presentdisclosure. This embodiment comprises three main components: a processor210, a sensing unit 220, and a head-mounted display (HMD) 230. Thesensing unit 220 may comprise a stereo camera pair 221, a distancesensor 222, and an illumination source 223. The processor 210 maycomprise camera calibration information in a memory module 211, and itmay be used to control a magnification setting of embodiments of thepresent disclosure, based on input from the user such as voice commands,button presses, or gestures, or other means of capturing the user'sintention. The processor 210 may receive information in the form of leftand right images from the stereo camera pair 221 as well as distancemeasurements from the distance sensor 222. The processor 210 may be usedto perform a transformation of the left and right images from the stereocamera pair 221 based on the camera calibration information and thedistance measurements, especially to make the images appear to the userin a way that when displayed, they cause the eyes to converge to anominal or actual working distance, and it may send the transformedimages for display to HMD 230. The processor 210 may filter distancemeasurements over time, and it may adjust settings of the distancesensor 222, stereo camera pair 221, illumination source 223, or HMD 230.For example, it may adjust an integration time or field of view of thedistance sensor 222 or an exposure time of the stereo camera pair 221 orillumination levels or spatial distribution of illumination source 223based on image information from the stereo camera pair 221 or distancemeasurements from the distance sensor 222 or other sources ofinformation, such as an ambient light sensor. The processor 210 mayadjust a focus setting of one or both cameras of the stereo camera pair221, and/or one or both the eyepieces of the HMD 230, and/or it mayreceive focus distance information from the stereo camera pair 221 andcompare it with distance measurements of the distance sensor 222.Furthermore, the processor 210 may be used to control and/or performoptical and/or electronic image stabilization. Distance sensor 222 maycomprise, for example, a time-of-flight sensor based on light or sound,or a sensor based on triangulation or capacitance, or any other knownmeans of measuring a distance. The function of distance sensor 222 maybe carried out by a stereo camera pair such as the stereo camera pair221 and processor 210 in the sense that distance information can becalculated from the stereo disparity between images obtained from acalibrated stereo camera pair.

The illumination source 223 may comprise one or more different kinds ofillumination sources, such as white LEDs designed with phosphors tocover a substantial portion of the visible spectrum, or LEDs or lasersused for fluorescence excitation, or multiple LEDs combined to form awavelength tunable illumination source, or incandescent or plasmasources, such as a xenon arc lamp, either present on the sensing unit220, or placed remotely but guided to the sensing unit 220 via a lightguide, or placed remotely and guided via free-space propagation to thesurgical wound. The processor 210 may pulse the illumination source 223in synchronization with the exposure interval of the stereo camera pair221 in order to achieve a shorter exposure time than would be possiblewith the same average illumination intensity but without pulsing; suchpulsing is a useful strategy to mitigate motion blur at highermagnifications. The processor 210 can control the angular extent orangular/spatial distribution of the illumination beam exitingillumination source 223, potentially as a function of distance measuredby the distance sensor 222, to match a field of view of the stereocamera pair 221, potentially as a function of the magnification of thedigital loupe system. Variation of the angular and/or spatial extentand/or distribution of the illumination can be accomplished in multipleways: by using a zoom optic in front of an LED; by using an array ofindividually addressable LEDs in front of a lens such that theillumination intensity profile at the surgical wound is controlled bythe intensity setting of each LED; or, by employing other forms oftunable beam shaping, for example, those developed by LensVector™. Theillumination source 223 can comprise multiple individually addressableLEDs of different wavelengths, with light mixed together and directed ina beam toward the subject. With such an arrangement, it is possible tocapture multispectral images of the subject by time-sequentialillumination with the different wavelengths, or even better forvideo-rate imaging, by time-multiplexing combinations of wavelengths, asin Park, Jong-Il, et al. “Multispectral imaging using multiplexedillumination.” 2007 IEEE 11th International Conference on ComputerVision. IEEE, 2007.

FIG. 3 depicts portions of a binocular head-mounted display of anembodiment of the present disclosure. A user's left and right eyes 301a,b look into corresponding near-eye displays/eyepieces 331 a,b, of thehead-mounted display with a fixed convergence angle 305. Thehead-mounting and support structure of the near-eye displays 331 a,b(such as, e.g., one or more of the head-mount embodiments describedbelow) permits the interpupillary distance (IPD) 303 of the displays tobe adjusted so the optical axes of the near-eye displays/eyepieces 331a,b (and the centers of the displays) line up with the optical axes 302a,b of the user's eyes 301 a,b, thus projecting the center of eachdisplay on the center of each corresponding eye's retina. A virtualimage 309 of the near-eye displays 331 a,b is set at a virtual imagedistance 304 corresponding to a nominal working distance of a user, bysetting a proper focusing distance between the eyepieces and displays ofnear-eye displays/eyepieces 331 a,b. The virtual image 309 at virtualimage distance 304 is also where the optical axes 302 a,b nominallyintersect when aligned with optical axes of the near-eyedisplays/eyepieces 331 a,b. Therefore, whether the user is lookingthrough the near-eye displays/eyepieces 331 a,b, or directly at anobject or work area at the nominal working distance, there is little orno change in ocular accommodation or convergence, facilitating aseamless, comfortable transition between the two views. Furthermore, aswill be explained later, the ergonomics of the digital loupecontemplated in the present disclosure are such that both the object orwork area and the near-eye displays can be put into the user's field ofvision simultaneously, a condition enabled by the transversedisplacement of the stereo camera pair with respect to the optical axesof the near-eye displays.

As described above with respect to the digital loupe system of FIG. 1,some head-mounted display systems employ a distance sensor and a stereocamera pair to obtain images for display on near-eye displays. FIG. 4depicts viewpoint frustums 401 a,b indicating orientations and angularfields of view of a stereo camera pair of a head-mounted digital loupesystem, with viewpoint frustum 411 of a distance sensor having anoptical axis 413 that nominally bisects the angle between optical axes403 a,b of the stereo camera pair. Optical axes 403 a,b correspond tothe centers of the fields of view of frustums 401 a,b. Optical axes 403a,b converge toward a point near the nominal working distance of thedigital loupe's user, such that an object at their nominal convergencepoint is also at or near the nominal convergence point of the opticalaxes of the user's eyes (i.e., the convergence point of optical axes 302a,b in FIG. 3). For example, with reference to FIG. 3, interpupillarydistance 303 may be 60 mm, and angle 305 may be 0.1 rad, so distance 304may be approximately 600 mm, corresponding to a nominal workingdistance. Thus, an object point depicted at the center of each near-eyedisplay 331 a,b appears to be located at a distance of 600 mm from theuser's eyes. With reference to FIG. 4, ideally optical axes 403 a,b ofstereo camera pair frustums 401 a,b nominally converge at this samepoint 600 mm from the user's eyes. In practice, there may be slightangular misalignments of these optical axes from their ideal positionsthat will be dealt with subsequently.

The camera frustums 401 a,b of the stereo pair may each have a field ofview 402 a,b that is larger than a field of view of near-eye displays331 a,b. Nominally, near-eye displays 331 a,b depict a magnified viewcompared to what would be seen by the unaided eye. For example, angularmagnifications in the range of 2× to 10× may be used. In someembodiments, the magnification may be approximately 1×, e.g., nominallyunmagnified. One way to effect this magnification is to select a portionof the fields of view 402 a,b of each camera for depiction on eachdisplay 331 a,b at an enlarged size (e.g., cropping and zooming). Assumewe select a portion of each field of view 402 a,b around the opticalaxes 403 a,b for display. As the magnification of the digital loupesystem increases, the displayed portion of each field of view 402 a,bshrinks around respective optical axis 403 a,b. At high magnification,an object may disappear from displayed portions of the fields of view402 a,b if the object is not located near the nominal intersection pointof optical axes 403 a,b. Also, if there are slight misalignments in theoptical axes 403 a,b, e.g., if they do not intersect, it may not bepossible to view a magnified object with single vision, as the magnifiedobject will be displaced differently when viewed by each eye 301 a,bbased on the exact misalignments of each optical axis 403 a,b.

The solution to both of these problems is to use information from thedistance sensor represented by frustum 411, with potentially adjustablefield of view 412, and optical axis 413, along with camera calibrationinformation regarding cameras represented by frustums 401 a,b, in orderto compute a transformation of the images from cameras represented byfrustums 401 a,b prior to cropping and zooming. For example, suppose anobject is located at distance 414 along the optical axis 413 of thedistance sensor. If cameras represented by frustums 401 a,b had opticalaxes directed toward this object, e.g., directed along axes 404 a,b,they would record this object in the center of their fields of view 402a,b and therefore it would be displayed at the center of each display331 a,b, providing comfortable single vision without issue. However,because the object does not appear in the center of the fields of view402 a,b, it may not be possible to comfortably view the magnified objectwithout diplopia or even at all through near-eye displays 331 a,b.

In order to remedy this, the system can compute a transformation ofimages from cameras represented by frustums 401 a,b that depends ondistance measurements from distance sensor represented by frustum 411and camera calibration information (stored, e.g., in the system's memorymodule 211 in FIG. 2) to make the images appear as if the detectedobject at distance 414 were measured at the center of the fields of view402 a,b, that is, as if the axes 404 a,b were also the optical axes ofthe cameras represented by frustums 401 a,b. To do this we makeextensive use of the pinhole camera model, a useful mathematicalabstraction for relating the position of points in a 3-dimensional (3D)object space, corresponding to the real world, to positions in a2-dimensional (2D) image space, corresponding to pixel coordinateswithin an image. The operations referenced herein, including cameracalibration to obtain camera matrices and affine transformations totransform images between viewpoints based on camera calibrationinformation, are available as software routines in most computer visionsoftware packages such as OpenCV. A convention routinely seen in suchsoftware packages for the operations referenced herein is the use ofhomogeneous coordinates and the mathematics of projective geometry. A 3Dobject point X can be written in 4D homogeneous coordinates, and a 2Dimage point y can be written in 3D homogeneous coordinates. Neglectingimage distortion (as it is known in the art how to deal with imagedistortion in this process), the mapping between object and image spacecan be performed by multiplying the object point X by a (3×4) cameramatrix in order to obtain the image point y. The camera matrix comprisesboth extrinsic parameters relating to camera position and orientation,and intrinsic parameters relating to focal length, optical center of theimage, and pixel size. It is known in the art how to obtain theparameters of such a camera matrix of both single- and multi-camerasystems, through a procedure known as camera calibration, for exampleusing routines available in OpenCV. Camera calibration information mayalso be obtained using, e.g., the process described in Zhang, Z., “AFlexible New Technique for Camera Calibration,” Microsoft CorporationTechnical Report MSR-TR-98-71 (Dec. 2, 1998). Both the camera matrix anda matrix representing the inverse transform—mapping from coordinates ina given image to coordinates in the real world, up to a scale factor—canbe obtained. The inverse transform is known only up to a scale factorcorresponding to the depth or distance of the object point away from thecamera. However, if this distance is known, then the object pointcorresponding to an image point of the camera can be unambiguouslydetermined, aiding in the registration of image points recorded fromdifferent camera viewpoints yet corresponding to the same object point.

A camera matrix can be decomposed into a matrix of its intrinsicparameters and a matrix of its extrinsic parameters, with the fullcamera matrix a product of these two. The matrix of extrinsic parameterscorresponds to a rigid transformation potentially comprising bothrotation and translation. Let us call the camera matrix for each camerai of the stereo pair represented by frustums 401 a,b W_(i), which can bedecomposed into intrinsic components C_(i) and extrinsic componentsH_(i) such that W_(i)=C₁H₁. The optical axes 403 a,b of camerasrepresented by frustums 401 a,b nominally intersect at a certain workingdistance, perhaps with slight misalignments relative to their designeddirections, as well as slight misalignments with respect to the centerof each corresponding image sensor. Assume that distance sensorrepresented by frustum 411 is at the origin of a 3D Cartesian coordinatesystem, and a distance measurement to an object under observation isreported as a point along optical axis 413 with homogeneous coordinatesX=(0,0,z,1)^(T). This point can be transformed to an image point withcamera matrix W_(i), e.g., y_(i)=W_(i)X. Image point y_(i) is now takento be the center of the image from camera i, thus cropping and zoomingof this image takes place around this new image center. After croppingand zooming and display of the image in the corresponding near-eyedisplay 331 a,b, the object point corresponding to the intersection ofdistance sensor optical axis 413 with the object under observation wouldappear at the center of each near-eye display 331 a,b.

Another way to transform the images from cameras represented by frustums401 a,b would be to assume that the entire object under observation isplanar and perpendicular to optical axis 413 at measured distance z fromthe distance sensor represented by frustum 411. Each image point (a, b,1)^(T) of an image from camera i, expressed in homogeneous coordinates,is associated via the intrinsic camera matrix with a ray that emergesfrom the origin of that camera and passes through a point expressed inthe camera's object-space coordinate system. This ray can be written(x′w, y′w, w)^(T), where the prime indicates we are in the camera'scoordinate system. This coordinate system can be transformed to thereference coordinate system of the distance sensor represented byfrustum 411 using the inverse of the extrinsic camera matrix. If weassume the object lies in the plane perpendicular to optical axis 413 atmeasured distance z, we can solve for parameter w at each image point toget the coordinates of the assumed object point corresponding to eachimage point. This procedure is equivalent to calculating theintersection of a ray, associated with an image point, and the assumedplanar object detected by distance sensor represented by frustum 411.For each camera i we can assign an ideal extrinsic camera matrix thataims the center of the camera toward the point X at measured distance zalong optical axis 413; in FIG. 4, this would correspond to theredirection of camera frustums 401 a,b along axes 404 a,b if distance zwere given by 414. We can transform image points to new image points, asif the camera were aimed at point X and assuming a planar object, bymultiplying object point coordinates corresponding to each image pointwith this ideal extrinsic camera matrix and then with the intrinsiccamera matrix. Although similar to the previous simpler procedure thattranslated a given image so its center lined up with a point alongoptical axis 413, this latter procedure is more general as it cancapture the full homography between the observed image from camera i andan image with the camera in an ideal orientation (e.g., aimed at pointX). However, assuming the ideal camera position and orientation issufficiently close to the actual camera position, there is not asignificant difference between the two procedures.

After completing the transformations enumerated in the above procedure,left and right images of an object or work area are displayed in, andcentered with respect to, the left and right eyepieces of a head-mounteddisplay, such as near-eye displays 331 a,b of FIG. 3. As the opticalaxes 302 a,b of these displays converge with angle 305 to a point atnominal working distance 304, which may be similar to the actual workingdistance, for example distance 414 of FIG. 4, the eyes 301 a,b will nothave to significantly change convergence to look directly at the objector work area versus viewing the object or work area through near-eyedisplays 331 a,b. Furthermore, it is possible for the processor 210 tovirtually (by translation of displayed images to the left and right) orphysically (by rotation of the near-eye displays 331 a,b) adjust theconvergence angle 305 of near-eye displays 331 a,b, such that when leftand right eyes 301 a,b look through near-eye displays 331 a,b, theyconverge to the actual working distance corresponding to a measurementfrom the distance sensor 222 represented by frustum 411. It is alsopossible for the processor 210 to virtually or physically change theconvergence angle 305 in proportion to the change in measured distanceto the object or work area under observation. Finally, it is possiblefor the processor 210 to change the focus state of near-eye displays 331a,b to cause the virtual image plane 309 to match or track the actualworking distance corresponding to measurements from the distance sensor222. In this way, no or minimal change in visual accommodation and/orvergence state of eyes 301 a,b would be needed to switch between viewinga subject directly, e.g., above or below near-eye displays 331 a,b, andthrough the near-eye displays 331 a,b.

It is a feature of the present disclosure that the distance sensorrepresented by frustum 411 may have a defined field of view 412 that maybe adjustable. Distance measurements may come from those objects thatare within the field of view 412 only. If this field of view is tied tothe magnification of the digital loupe system, then as the magnificationof the digital loupe increases, the field of view 412 of the distancesensor represented by frustum 411 can decrease. This is to ensure thatthe field of view 412 of the distance sensor represented by frustum 411matches (or corresponds to) the field of view displayed to the userthrough near-eye displays 331 a,b. The VL53L1X distance sensor fromSTMicroelectronics, Inc., a LiDAR time-of-flight sensor, affords such afeature of adjustable field of view. However, changing the field of viewaffects the amount of light collected in a given distance measurement,affecting measurement precision, and individual measurements may not besufficiently precise to begin with, so some form of temporal filteringof the distance measurements is desired. The distance sensor representedby frustum 411 can be calibrated to ensure accuracy of its distancemeasurements under working conditions. Also, camera calibrationinformation (e.g., orientation and position) can be referenced tocalibration information of the distance sensor represented by frustum411, e.g., the coordinate system defined by the position and orientationof the distance sensor represented by frustum 411.

In some embodiments, it may be preferable to have a distance sensor witha narrow, collimated beam, such as a laser-based time-of-flight distancesensor like the TF-Luna distance sensor from Benewake Co., Ltd., sothere is minimal ambiguity about the actual distance measured within thefield of view. Generally, time-of-flight sensors report the measureddistance based on a statistic such as the mean time-of-flight of allcollected photons. If the collected photons form a histogram of photoncounts vs. distance that is bimodal (for example, if the active area ofthe distance measurement includes a distinct edge with a foregroundobject and a background object), the mean will be between the two peaksand thus the distance reported will not correspond to the center ofeither peak. Therefore, the optics of the distance sensor can beconfigured to have a narrow beam, minimizing the probability ofencountering an ambiguous distance measurement scenario.

Additional possibilities are enabled if the distance sensor representedby frustum 411 is an imaging distance sensor that provides a spatiallyresolved map of points, or a point cloud, across its field of view.Consider the previous case concerning an assumed planar object atmeasured distance z along optical axis 413 and perpendicular to thataxis. With spatially-resolved distance information, we can relax theassumption that the object is planar. The point cloud reported by theimaging distance sensor represents points on the surface of the object,and these points can be mapped to the camera coordinate system toassociate each image point with an object surface point. The implicationis that for each point in the image, we can find the precise objectpoint in our reference coordinate system. Thus, we can reproject theobject points of a given image using a new, virtual camera matrix, toview them as if they were imaged through a virtual camera that may havea different position, orientation, focal length, etc. For example, thesensing unit 120 is worn on the forehead of surgeon 100, but the headset130 is worn naturally in front of the eyes. We can reproject the imagesderived from sensing unit 120 as if they were imaged by cameras at thepositions of the eyes of the surgeon 100, especially if the relativeposition and orientation of the cameras and the surgeon's eyes is knownat least approximately. This way, the effective viewpoint of the sensingunit 120 is the same as for the surgeon 100, reducing or eliminatingparallax with respect to the viewpoint of the surgeon 100. Even withoutan imaging distance sensor, it may still be useful to perform thisoperation to remove the average parallax across the image, which couldbe done by once again assuming the object is planar at a distance zalong the optical axis 413, and then reprojecting those assumed objectpoints onto the viewpoint of the surgeon 100.

Returning to FIG. 2, note that processor 210 may be configured to updatecamera calibration information stored in memory 211 during operation ofa digital loupe system, for example by going through a cameracalibration routine as described in the above-referenced publication byZhang. Alternatively, the processor 210 can identify similar featuresbetween the cameras of the stereo pair 221 and adjust camera calibrationinformation 211 such that when the processor 210 transforms images ofthe stereo pair 221 using either a translation or a full homography,these similar features show up in similar locations for each eye of thebinocular head-mounted display 230. This could be done using aself-calibration technique as described in Dang, T., et al., “ContinuousStereo Self-Calibration by Camera Parameter Tracking,” IEEE Trans. ImageProc., Vol. 18, No. 7 (July 2009). This would be important for slightmisalignments of the optical axes of the stereo pair 221 that mightaccrue over time during operation of the digital loupe system.

In another embodiment of the present disclosure, a multi-channel imageris provided that combines an array of multiple single-channel imagersand uses an imaging depth sensor to remove parallax from the multiplesingle-channel imagers, such that the multi-channel image appears to bederived from a single camera or viewpoint. The process of mapping oneviewpoint to another may be identical to that used for the previouslydescribed embodiment of the present disclosure. For example, themulti-channel imager can include a processor configured to store cameracalibration information relating to at least two cameras, wherein thecalibration information is defined in a coordinate system relative to animaging distance sensor of the system. A processor of the multi-channelimager may be configured to receive image signals from the cameras anddepth information from the imaging distance sensor, and use the depthinformation and the camera calibration information in order to correctfor parallax between the cameras, thus providing a multi-channel imagethat appears to originate from a single viewpoint. Some examples ofmulti-channel imagers are hyperspectral imagers or Stokes imagingpolarimeters. Certainly, as in the prior art, an imaging depth sensorcan be used to combine images from different modalities—for example, US2018/0270474 A1 teaches that depth information can be used to registerimages acquired with diverse intraoperative optical imaging modalities,such as NIR fluorescence, color RGB, or hyperspectral imaging using atunable liquid-crystal filter or a mechanical filter wheel. But so farno one has envisioned using depth information to enable asingle-modality multi-channel imager. It is a conceptual leap from theprior art to consider that a multi-channel optical imager could becollectively formed out of an array of single-channel imagers arrangednominally in a plane transverse to their lines of sight, in conjunctionwith an imaging depth sensor that provides sufficient information toremove effects of parallax from the different positions of the imagerarray. The output of such a system would comprise a multi-channel imagecube as if obtained from a conventional multi-channel imager, that is,from a single viewpoint.

Such a multichannel imager could be combined with the digital loupesystem of the present disclosure to simultaneously provide otherintraoperative optical imaging modalities within the magnified view ofthe digital loupe system. For example, the array of sensors of theenvisioned multi-channel imaging system could comprise multipleindividual spectral bands, such that taken together with parallaxremoved, the output would comprise a multispectral or hyperspectralimage. This hyperspectral image can be analyzed and compared to priorinformation to determine regions of the surgical wound 110 comprisingcancerous tissue to be resected. An image can be formed indicating theprobability of cancerous tissue at each pixel location. This image canbe combined, as an overlay or using known image fusion techniques, withthe magnified image presented in the display 130 of the digital loupesystem, so a surgeon 100 has a more precise map of where to resecttissue than from the magnified image alone.

Similarly, the channels of the multi-channel imager could eachcorrespond to an independent Stokes polarization component. Thus, themulti-channel imager could comprise a Stokes imaging polarimeter. AStokes imaging polarimeter would be a useful addition to a digital loupebecause it could be used to provide images with reduced glare, eitheralone or by modifying the polarization of the illumination. If used incombination with circularly polarized illumination, the Stokespolarization image can potentially be used to visualize birefringentstructures such as nerves, as described in Cha et al., “Real-time,label-free, intraoperative visualization of peripheral nerves andmicro-vasculatures using multimodal optical imaging techniques”,Biomedical Optics Express 9(3):1097.

Other embodiments of the digital loupe system capture enhancements withrespect to the prior art. For example, as mentioned in the Backgroundalong with the associated drawbacks, U.S. Ser. No. 10/230,943 B2 teachesa type of digital loupe with integrated fluorescence imaging such thatwithin one sensor, both NIR (fluorescence) and visible (RGB) light arerecorded, with a modified Bayer pattern where pixels in both visible andinfrared bands can be tiled on the same sensor. The stereo camera pairof the present disclosure could comprise one or more such sensors. Alimitation of such a sensor is that the same exposure, gain, and othersettings are used for the NIR and visible light as they are imagedsimultaneously. However, certain modern image sensors have ahigh-dynamic-range (HDR) capability that successively takes multipleexposures with different exposure durations. One could take advantage ofcombining HDR with such an RGB-NIR sensor in order to separatelyoptimize imaging conditions, e.g., exposure duration, for both visibleand near-infrared light.

Some aspects of the present disclosure aim to enhance the userexperience of a digital loupe system. For example, it may be desired tosoften the edges of the displayed image in each eye, e.g., with digitalvignetting, in order that the eye is not drawn to the sharp edges of theimage.

The digital loupe system may include an ambient light sensor thatdetects the spectrum and/or intensity of the ambient light. It is wellknown that ambient light can affect a viewing experience, so ameasurement of ambient light can be used to adjust, for example, thewhite point and the brightness setting of the head-mounted displays ofthe digital loupe system.

It may be useful to present the image in the digital loupes with aspatially variable magnification. For example, a center rectangularportion of the image in each near-eye display, perhaps covering an areaextending 20% across each dimension of the field of view of eachdisplay, can be displayed with a magnification substantially higher thanthe surrounding portion. If this high magnification were used across thewhole image, the user may lose context of portions of the objectsurrounding the displayed portion. However, with spatially variablemagnification, it is possible to achieve both high magnification andpersistence of context simultaneously.

The processor of a digital-loupe system can comprise the most generalcolor-substitution algorithm, which is a 3-dimensional look-up tablethat substitutes a given color for another. It is known that the eye'sresponse or sensitivity to different colors and intensities of lightdiffers substantially from that of a standard color camera. For example,the eye is most sensitive to changes in light intensity at greenwavelengths, and is less sensitive to changes in light intensity at redwavelengths and blue wavelengths. It is likely then that there is a lossof useful information between a color image as it is recorded and whenit is displayed to a user. There are many red hues expected from imaginga surgical operation, primarily due to the presence of hemoglobin inblood, as well as other bodily pigments. Not only is the human eye lesssensitive to red wavelengths, but typical electronic displays may havetrouble reproducing the saturated reds that images of blood comprise, asthey may be outside of the display gamut. In either case, it may beadvantageous to shift red colors, especially saturated red colors,toward the green (e.g., make them yellow) in order that the eye candiscriminate between more subtle variations in red-colored tissue. Ineffect, this increases the amount of perceptual information available tothe user. This can easily be done with a 3-dimensional look-up table.Color substitution may also be dynamic or may be determined by analgorithm which may utilize machine learning.

Ergonomic enhancements are also provided in various embodiments of thepresent disclosure. FIG. 5 shows a frontal projection of a human head500 with forward gaze. Note that this disclosure is not limited to aconfiguration that requires a forward gaze of a user; for example, auser might have a downward gaze. Vertical lines 510 intersect withhorizontal line 511 at the pupil of each eye. Circles 531 and 532 arecentered approximately with respect to the pupils such that an objectwithin circle 531 will appear closer to the center of vision of thehuman depicted in FIG. 5 than an object within the circle 532 but notwithin the circle 531. Objects outside of the circles 532 will eitherappear within the human's peripheral vision (i.e., only at the edge ofthe human's vision) or will not be seen at all. Vertical lines 521intersect the frontotemporales of the human head 500 to define regions522 lateral to the frontotemporales. The frontotemporales are the mostanterior points of the temporal ridges on either side of the frontalbone of the skull; the temporal ridges mark a sort of transition pointbetween more vertically sloped portions of the skull on the lateralside, and more horizontally sloped portions on the medial side. Region512 is medial and superior to the pupils, and extends vertically toabout the top edge of the human's peripheral vision, approximately inline with the eyebrow ridge of head 500, or to the glabella, which isthe point between the eyebrows.

Ocular supports of the prior art, when viewed in frontal projection uponthe head 500, generally encroach upon, intersect with, or are mountedwithin region 512 and/or regions 522. For example, glasses-like supportsutilize temple pieces that are supported by the ears within regions 522.Also, prior binocular head-worn magnifying loupes comprise a pair ofsimple magnifiers mounted in a visor that attaches to a headband on thesides of the head, lateral to the frontotemporales. Front-lens-mountedloupe systems or flip-up mounted systems typically have a support armthat descends from above within region 512 when viewed in frontalprojection.

When viewed in a frontal projection upon head 500, ocular supportsystems or support arms of the present disclosure may support an ocularin a line of sight of the eye, then extend laterally, posteriorly, andsuperiorly (e.g., at least radially outward with respect to circles 531and 532) while avoiding intersection with region 512, then extend to ahead engagement member at positions that are medial to regions 522.Secondary support arms may intersect regions 512 and/or 522, for exampleto link together two oculars that are supported via primary support armswhich follow the above-described pattern. A secondary support arm thatlinks two oculars and crosses through region 512 can still besubstantially outside of the peripheral vision of the user if it isrouted in such a way that from the point of view of the user that itrests primarily behind the apparent field of view of the oculars. It isalso beneficial if the image viewed through the oculars extends to theedge of the ocular. Although this approach makes the image edge blurrybecause the ocular edge is near to the eye and not in focus, thepresence of this blurry image edge within the user's field of viewobscures the ocular support arms even further, making the image appearas if it floats in front of the eye with minimal visible support. Also,the blurring at the edge of the image is useful to prevent the eye frombeing drawn to a sharp image edge, which could otherwise disturbbinocular vision by providing conflicting binocular cues when twooculars are used in a binocular head-mounted display.

Specific head mounting systems for oculars employing ocular support armsthat meet the general criteria as enumerated above are described indetail further below. They are preferable to ocular support systems witha primary support arm that descends through region 512 because they donot create the same uncomfortable sensation of having somethingimmediately in front of the face. Extending the proximal ends of theocular support arms to positions medial to the frontotemporales enablesthe head-mounted ocular support systems of this disclosure toaccommodate different user head widths, which is easier to do if theproximal ends of the support arms extend to a head engagement member ator near the top of the head rather than to the sides of the head. Insome embodiments, the two support arms are separate structures supportedby the head engagement member. In other embodiments, the two supportarms are part of a unitary structure supported centrally by the headengagement member and extending distally from the central support pointto their respective oculars or ocular support structure.

FIG. 6 shows a plot 600 of the extent of the field of vision 606 for aleft eye of a subject. Vertical line 601 and horizontal line 602intersect at the center of vision, corresponding to the fovea. Contours603, 604, and 605 represent particular angular deviations away from thecenter of vision, each one a greater deviation from the center than theprevious. For example, contour 603 represents a deviation of 10 degreesfrom the center of vision, contour 604 represents a 30 degree deviation,and contour 605 represents a 60 degree deviation. Regions of vision canbe specified to lie within one of four quadrants. Those on the same sideof vertical line 601 as the subject's nose are labeled “nasal”, whereasthose on the same side of vertical line 601 as the subject's left templeare labeled “temporal”. Likewise, regions above horizontal line 602 arelabeled “superior” whereas those below horizontal line 602 are labeled“inferior”. The four regions are thus the nasal superior 610, temporalsuperior 611, temporal inferior 612, and nasal inferior 613. The outlineof an ocular 620 is shown as centered upon the center of vision, thoughthis is only a nominal position, and other positions near the center ofvision are anticipated. Ocular 620 is supported by ocular support arm621.

Embodiments of the present disclosure comprise an ocular, such as ocular620, supported by an ocular support arm, such as support arm 621, thatattaches to the ocular in such a way as to avoid occluding vision in thenasal superior region 610. The support arm has a more distal portionextending laterally beyond the ocular support location, a more proximalportion extending medially toward the head engagement member, and acentral portion that extends between the distal and proximal portionsbeyond, or nearly beyond, the periphery of the user's vision. In someembodiments, the support arm may have multiple segments that are movablewith respect to each other to change the position of the ocular itsupports and to adjust the system to fit the user's head. Ocular supportarms as described herein, from the point of view of the user, have thesame advantages as those described with reference to FIG. 5: minimalobscuration of peripheral vision, especially in the sensitive areabetween and above the eyes, and the ability to adapt to a range of headwidths.

FIGS. 7A-C depict an embodiment of a digital loupe system 700 as wornupon a user's head 701. The head mounting system of this embodiment maybe used to support oculars other than digital loupe oculars. Portions ofthis head mounting system may also be used to support a single ocularusing, e.g., a single ocular support arm and associated structure. FIG.7A depicts a perspective view, FIG. 7B depicts a side view, and FIG. 7Cdepicts a front view. This embodiment comprises an adjustable binoculardisplay and support structure 710 and a stereo camera pair 720 mountedon a head engagement member 730 on the user's head 701. The adjustablebinocular display and support structure has a pair of oculars 711 a and711 b supported by adjustable support arms that minimize interferencewith the user's vision, as described below. The stereo camera pair 720is mounted in a housing 702 with an adjustable declension angle viarotational hinge 721 so that the cameras 722 a,b in the camera pair 720can be pointed in the desired direction toward, e.g., an object or workarea. In addition to the stereo camera pair 720, a distance sensor 723and an illumination source 724 are disposed in housing 702. The cameras722 a,b, distance sensor 723 and illumination source 724 all haveoptical axes that converge at a nominal working distance of the user,such as 50 cm. As described with respect to FIGS. 1-4 above, the cameras722 a,b and distance sensor 723 are controlled by a controller (notshown) to display on oculars 711 a,b images of, e.g., a work area orobject for viewing by the user wearing the device.

In this embodiment, the oculars 711 a and 711 b are supported by asegmented support arm structure which extends proximally from distalocular support locations to the periphery of the user's vision byextending laterally, posterially, superiorly and medially beforecoupling to a head engagement member 730 in a position medial to thefrontotemporales. In embodiments, the support structure includes anoptional display bar to which the oculars are movably attached as wellas a pair of support arms, which may comprise multiple articulationsthat allow for the adjustment of the lateral position of each ocular,e.g., to adapt to different user interpupillary distances; coupledadjustment of the vertical declension angles of the oculars; coupledadjustment of the vertical position of the oculars; and coupledadjustment of the eye relief distance of the oculars. Furthermore, theclearances between the support arms and the sides of the head may beadjustable.

Specifically, oculars 711 a and 711 b are both coupled to display bar712 with slidable coupling mechanisms in order to adjust interpupillarydistance. Display bar 712 forms an ocular support arm that is secondaryto side support arms 715 a,b, and is primarily obscured from theperspective of the user by oculars 711 a,b, which may display imagesthat extend at least to the edges of the oculars. A convergence angle ofthe oculars can be maintained independent of their sliding position, oradjusted with an additional articulation (not shown) that would rotateeach ocular inward with respect to the other. Display bar 712 extendslaterally from the oculars to connect to distal ends of side supportarms 715 a and 715 b via hinges 713 a,b and hinges 714 a,b. Display bar712 can rotate about hinges 713 a,b to adjust a declension angle of theoculars. The declension angles of both oculars adjust together in thismanner, avoiding divergence and thus avoiding double vision. Hinges 714a,b permit side support arms 715 a,b to be moved toward and away fromthe side of the user's head.

In the embodiment shown in FIGS. 7A-C, side support arms 715 a and 715 beach have three straight segments connected by an angle connector 703a,b and a sliding connector 716 a,b. In other embodiments, the sidesupport arms may be unitary components that have straight and/or curvedportions. Sliding connectors 716 a,b enable adjustment of the verticalheight of oculars 711 a,b with respect to the user's head 701 bychanging the effective height of side support arms 715 a,b, i.e.,changing the distance side support arms 715 a,b extend inferiorly fromthe head engagement member. The side support arms 715 a,b arerotationally connected via hinges 717 a,b to a top support arm 718,which is coupled to the head engagement member 730 via rotational hinge719. When the head engagement member is engaged with the user's head,rotational hinge 719 is medial to the user's frontotemporales. Likehinges 714 a,b, hinges 717 a,b permit side support arms 715 a,b to bemoved toward and away from the side of the user's head. The rotationalaxes of hinges 714 a and 717 a are nominally collinear, and therotational axes of hinges 714 b and 717 b are nominally collinear, toenable movement of the side support arms 715 a,b to adjust clearancebetween the side support arms and the side of the user's head. Eyerelief, or the distance from oculars 711 a,b to the user's face, isprimarily adjusted via rotation of top support arm 718 about hinge 719,which results in movement of side support arms 715 a,b and display bar712 toward or away from the user's face. When the head engagement member730 is engaged with the user's head, display bar 712 extends laterallyfrom oculars 711 a,b to side support arms 715 a,b, and side support arms715 a,b extend posteriorly and superiorly from hinges 713 a,b inpositions at or beyond the periphery of the user's field of vision.Support arms 715 a,b may also extend laterally if they have been rotatedaway from the user's head about hinges 714 a,b and hinges 717 a,b. Topsupport arm 718 extends medially from its connections to side supportarms 715 a,b to the head engagement member 730. Thus, this configurationenables the support arms to extend from the oculars to their connectionto the head engagement member medial to the user's frontotemporaleswithout extending through a region of the user's face medial andsuperior to a center of the user's eyes and inferior to the user'sglabella.

FIGS. 8A-8C show multiple articulation states of the embodiment of thedigital loupe system 700 as shown in FIGS. 7A-C, with the forward viewof the embodiment as shown in FIG. 7C reproduced for reference in FIG.8A. FIG. 8B shows the system 700 adjusted to give the user a greaterinterpupillary distance with respect to the state shown in FIG. 8A,which can be effected by sliding the oculars 711 a,b along the displaybar 712. FIG. 8C shows the system 700 with a greater clearance betweenside arms 715 a,b and the sides of the wearer's head 701 than the stateshown in FIGS. 8A and 8B; this state involves a change in state ofrotational hinges 714 a,b and 717 a,b.

FIGS. 9A-9D show further multiple articulation states of the embodimentof the digital loupe system 700 as shown in FIGS. 7A-C, with the sideview of the embodiment as shown in FIG. 7B reproduced for reference inFIG. 9A. FIG. 9B shows the system 700 adjusted to give the user anincreased camera declension angle, effected by a rotation of housing 702about hinge 721. FIG. 9C and FIG. 9D both show states in which theoculars of system 700 have decreased declension angles, with theconfiguration of FIG. 9D having less declension and more eye relief forthe user than the state of FIG. 9C. Both of these states are reached byrotation of display bar 712 about hinges 713 a,b, adjustment of sidesupport arms 715 a,b via slides 716 a,b, and rotation of upper supportarm 718 about hinge 719.

It should be appreciated that the different articulation states of FIGS.8A-8C and 9A-9D are representative samples from a continuum ofarticulation states, and that surgeons can choose an articulation statethat provides the best fit and ergonomics in terms of multiple factors,by intuitively adjusting the position and declension of the oculars. Oneway to capture the notion of “intuitive” in terms of adjustment of theposition and declension of the oculars is the following. Each operatingposition as shown in FIGS. 8A-8C and 9A-9D comprise a particular stateof each of the points of articulation, such as slides and hinges. Thestate of each articulation exists in a one-dimensional continuum, thusoperating positions comprise points in a multidimensional space that isthe product of each one-dimensional articulation range. An adjustmentcan be called intuitive if adjusting between two operating positionscorresponds to traversing a straight line in this multidimensionalspace. Ideally, operating positions are uniquely defined by one point inthis configuration space.

The flexibility afforded by the various articulations proffers multipleadvantages, one of which is the ability to provide optimal ergonomicsfor a complete range of head shapes and sizes as well as operatingstyles. The interpupillary distance of oculars 711 a,b can be adjustedto match that of any surgeon. Depending on how the supporting headengagement member 730 rests on the surgeon's head 701, the oculars 711a,b may differ in position relative to the surgeon's eyes even if allthe articulations are in the same state—e.g., same slide position forsliding articulations, and same rotational position for rotationalarticulations. Therefore, the adjustment ranges of both the verticalposition and the eye relief can be made large enough to take intoaccount both the variation in how the head engagement member 730 mightbe supported on the surgeon's head 701, as well as a range of headshapes, sizes, and hairstyles (different hairstyles may cause the headengagement member 730 to sit differently on the surgeon's head 701).Also, a wider face can be accommodated by spreading out the side supportarms 715 a,b, as in the state shown in FIG. 8C versus the state shown inFIG. 8B.

Even for a given surgeon, the articulations confer flexibility ofoperating style. The adjustable height and declension of the oculars 711a,b, combined with the adjustable declension of the stereo camera pair720, allows the surgeon to set up an operating posture whereby she canview the surgical field or work area directly with her eyes, and thenwith only a small eye rotation, concurrently view the magnified, oraugmented, surgical field as displayed in the oculars 711 a,b. Thesurgeon can adjust the height and declension of the oculars 711 a,bdepending on whether she chooses to view the unmagnified surgical fieldabove the oculars with a slight upward eye rotation, or below theoculars with a slight downward eye rotation. The surgeon can choose tooperate in a standing position or a sitting position by simpleadjustment of the declension angle of the stereo camera pair 720 toredirect it toward the surgical field. If standing, it may be preferableto have a direct view of the surgical field below the oculars as opposedto above the oculars, as this maintains a more vertical cervical spine,thus decreasing the complications associated with forward head posture.The optical axes of the stereo camera pair 720 and the optical axes ofthe oculars 711 a,b can be adjusted to converge together at a nominalworking distance of a user, or they can be adjusted to diverge, suchthat the user can assume a more upright head position while stillviewing a work area that is directed downward, by increasing thedeclension of the stereo camera pair 720.

A given surgeon may choose different articulations of side arms 715 a,bin order to accommodate various eyeglasses or protective eyewear or faceshields. It is also possible to incorporate a face shield directly intothe frame 710 by attaching one or more transparent windows to the ocularsupport arms. The face shield can be constructed so as to leave theoptical paths from the camera 720 to the surgical field, and from theuser to the oculars 711 a,b, unobstructed. It can also have segmentsattached to the side arms 715 a,b in order to provide wraparoundprotection. It can be detached from the frame to be replaced with adifferent kind of face shield, for example one that incorporates laserfilters to protect the eyes from different laser wavelengths that may bein use during the operation.

Features of head engagement member 730 are shown in FIGS. 10A-10B. Sucha head engagement member has multiple inventive features that are usefulespecially to support the stereo camera pair and oculars of a digitalloupe system, such as the digital loupe systems described above.Firstly, the head engagement member must accommodate ranges of headlength, head circumference, slope and curvature of the front of thehead, and slope and curvature of the back of the head. Also, it mustprovide a stable mounting platform for the stereo camera pair and theoculars that is rigidly and closely coupled to the skull of the surgeon,such that head movements of the surgeon directly translate to movementsof these subsystems, without amplification or oscillation caused by longand/or finitely stiff lever arms.

Head engagement member 730 has an adjustable circumferential headband1001 and an adjustable superior strap 1031. Back channel 1033 receives apair of flexible bands including 1023 a, which can be adjusted in lengthusing actuator 1034, for example with a rack and spur gear mechanism, toadapt to variations in head circumference. Flexible support 1032suspends the back of the head engagement member 730 over the back of thewearer's head, but it is conformable and flexible in order to adapt todifferent curvatures and slopes of the back of the head. The flexiblebands including 1023 a comprise a rotational attachment including 1022 athat allows the angles of flexible headband extensions 1021 a,b tochange relative to the angles of the flexible bands including 1023 a.This is to accommodate differences in relative slope of the front andback of the head, as the flexible extensions 1021 a,b are rigidlycoupled to headband pieces 1020 a,b, which are made out of a more rigidmaterial. Adjustable strap 1031 adapts to different head lengths and canbe used both to help set the height at which center piece 1010 sits onthe head, as well as to transfer weight (downward force) from objectsmounted to it more toward the back of the head. Center piece 1010 hasmounting points 1040 and 1041 for various attachments, such as a stereocamera pair and/or a support frame for oculars, as described above withrespect to FIGS. 7A-C. Piece 1030 serves as an attachment point forstrap 1031. Piece 1010 is designed to stably engage the user's head inorder to support and maintain the stability of the stereo camera pairand ocular support subsystems attached to it. Note that piece 1010 issupported via tension from three directions to engage it with the user'shead, that is, from the two sides and from the top.

Piece 1010 has a toroidal curvature that approximates the curvature ofthe average front of the head. It can include a thin layer of conformalmaterial, such as gel or foam, that rests upon the head, withoutsignificantly decoupling it from motions of the head. Pieces 1020 a,balso have a toroidal curvature that approximates the curvature of theaverage head where they would be located on such a head. They can alsoinclude a thin layer of conformal material as described above. Theselayers of conformal material serve to better match the shape of thewearer's head. Flexible couplings 1011, 1012, shown here as rotationalhinges, between the side pieces 1020 a,b and the center piece 1010,allow the combination of pieces to better match the curvature of awearer's head over a larger distance, where deviations between thecurvature of an average head and of the wearer's head would become moreapparent. Thus, the segmented nature of the front of the head engagementmember allows a larger surface to be rigidly and closely coupled to theuser's head than a single piece could be, providing more support fordistributing the weight of attachments, and thus more comfort.

It will be appreciated by those skilled in the art that depending ondesign intention, not all articulations of digital loupe system 700,including its head engagement member 730, are needed. The articulationscould also be designed in different ways to achieve the same or similardegrees of freedom, and the support point for the ocular frame could bemoved forward or backward on the skull, while still achieving all theaims of the present disclosure. FIGS. 11A-11D depict some aspects of adifferent embodiment of a digital loupe 1100 of the present disclosurein perspective view (FIG. 11A), front view (FIG. 11B), and side views(FIGS. 11C-D). The head mounting system of this embodiment may be usedto support oculars other than digital loupe oculars. Portions of thishead mounting system may also be used to support a single ocular using,e.g., a single ocular support arm and associated structure.

FIG. 11D depicts a different articulation state than the states in FIGS.11A-C. Oculars 1105 a,b are movably supported by display bar 1104 (e.g.,via sliding connections permitting adjustment of the distance betweenthe oculars, as described above), which is rotationally coupled viahinges 1106 a and 1106 b to a unitary, ram's horn-shaped support arm1101.

A housing 1190 for a stereo camera pair 1192 a,b is mounted on a centerpiece 1110 of a head engagement member 1140. A distance sensor (notshown) may also be disposed in housing 1190, as described with respectto the embodiments above. As in the embodiment of FIGS. 10A-B, centerpiece 1110 of head engagement member 1140 is designed to stably engagethe user's head in order to support and maintain the stability of thestereo camera pair and ocular support subsystems attached to it. Piece1110 has a toroidal curvature that approximates the curvature of theaverage front of the head. It can include a thin layer of conformalmaterial, such as gel or foam, that rests upon the head, withoutsignificantly decoupling it from motions of the head. Side pieces 1120a,b, of the head engagement member 1140 connect to center piece 1110 viaflexible couplings 1111 and 1112 (e.g., rotational hinges). Side pieces1120 a,b of the head engagement member 1140 also have a toroidalcurvature that approximates the curvature of the average head where theywould be located on such a head. They can also include a thin layer ofconformal material as described above. These layers of conformalmaterial serve to better match the shape of the wearer's head. Headengagement member 1140 may also have a headband and/or a superior strap,such as shown in FIGS. 10A-B.

A central portion of support arm 1101 connects to center piece 1110 ofthe head engagement member 1140 via a rotational hinge 1103 and a slider1102 to achieve positional degrees of freedom for the support arm 1101and the oculars supported by it in the vertical and eye reliefdimensions. When the head engagement member 1140 is engaged with theuser's head, rotational hinge 1103 and slider 1102 are medial to theuser's frontotemporales. The oculars 1105 a,b are supported by a movabledisplay bar 1104, and the oculars connect to display bar 1104 in amanner that permits the distance between the oculars to be adjusted. Asin the prior embodiment, together the display bar and support arm 1101extend posteriorly, superiorly and medially from the ocular supportpositions. In this particular embodiment, display bar 1104 extendslaterally and posteriorly from the oculars 1105 a,b, and the two sidesof support arm 1101 extend from their connections to the display bar1104 in a three-dimensional curve inferiorly, posteriorly, andlaterally; then superiorly, posteriorly and laterally; and finally,superiorly and medially toward hinge 1103 and slider 1102 of the headengagement member in positions at or beyond the periphery of the user'sfield of vision. Thus, this configuration enables the two sides of theunitary support arm to extend from the oculars to the connection to thehead engagement member medial to the user's frontotemporales withoutextending through a region of the user's face medial and superior to acenter of the user's eyes and inferior to the user's glabella.

FIG. 11D illustrates an articulation state differing from that in FIG.11C in that the oculars 1105 a,b are higher and closer to the eyes yetstill within a line of sight of the eyes. This is accomplished with adifferent articulation state of hinge 1103, a different state of slide1102, and a different state of hinges 1106 a,b. Display bar 1104 andeach of the two sides of unitary support arm 1101 extend laterally,posteriorly and superiorly (more specifically, inferiorly, posteriorly,and laterally; then superiorly, posteriorly and laterally; and finally,superiorly and medially) from the ocular 1105 a or 1105 b to beyond theedge of the user's peripheral vision, while avoiding the part of theface medial and superior to the pupils and below the glabella, beforeultimately extending medially toward the center piece 1110 to besupported on top of the head, medial to the frontotemporales. The ram'shorn shape of the support arm 1101 is such that the wearer can still useglasses or face shields, even for the widest faces, yet it restsprimarily outside of the user's peripheral vision. Note that in FIGS.11A-11D, 12A-12D, and 13A-13D the full support headband is not shown.

It should be clear that through considering variations of the shape ofthe support arm 1101, the mounting point proximal to the head could bemore toward the back of the head or more toward the front of the head. Acombination of two articulations at the mounting point, sliding and/orrotating, depending on the exact mounting position as well as otherdesign considerations, could provide vertical and eye relief positioningof the oculars. The articulations for the different adjustments couldalso comprise slides and/or hinges on the support arm. For example, withrespect to the embodiment of FIGS. 7A-C, the slides 716 a,b of supportarm 710 generally provide a vertical position adjustment for theoculars, but if the mounting point of the support arm is on the back ofthe head, similar slides can be used to adjust eye relief distance,whereas a rotational pivot point would provide primarily verticaladjustment capability. This kind of adjustment mechanism could beapplied to the embodiment of FIGS. 11A-D. However, a mounting pointtoward the front of the head, as shown in FIGS. 7A-C, is generallypreferable, as this provides a shorter, and hence more stable, supportstructure. Another way to adjust the interpupillary distance would be tohave a sliding mechanism that allows adjustment of the width of theocular support structure, for example, lengthening both display bar 1104and support arm 1101 at their midpoint.

FIGS. 12A-12D depict an alternative embodiment of an ocular supportstructure supporting, e.g., a digital loupe system. The head mountingsystem of this embodiment may be used to support oculars other thandigital loupe oculars. Portions of this head mounting system may also beused to support a single ocular using, e.g., a single ocular support armand associated structure. In this embodiment, head engagement member1210 has a shape adapted to fit a human head. As shown, head engagementmember 1210 support a stereo camera pair 1292 a,b. Rings 1220, 1221, and1222 provide a connection to a headband and superior strap (not shown)to hold head engagement member 1210 against the user's head, such asshown in FIGS. 10A-B. A vertical slide 1202 and a hinge 1203 support aunitary support arm 1201 and can be used to adjust respectively theheight and eye relief of oculars 1205 a,b supported by the support arm1201. A display bar 1204 supports the oculars 1205 a,b, and a slidingconnection between oculars 1205 a,b and display bar 1204 allowsadjustment of the oculars to accommodate a range of interpupillarydistances, as described above. Hinges 1206 a,b between display bar 1204and support arm 1201 allow for adjustable and coupled declension angleof the oculars. FIG. 12D depicts a different articulation state than theviews of FIGS. 12A-C, in which vertical slide 1202 and hinge 1203 havebeen adjusted to provide a more horizontal line of sight with more eyerelief. Together, the display bar 1204 and the two sides of support arm1201 extend from the oculars posteriorly, then laterally, superiorly,and medially in a partial rectangle shape to hinge 1203, which supportsarm 1201, on to beyond the edge of peripheral vision, while avoiding thepart of the face medial and superior to the pupils and below theglabella, before ultimately being supported on top of the head, medialto the frontotemporales.

FIGS. 13A-13D provide four views of yet another embodiment of a digitalloupe system according to the present disclosure. The head mountingsystem of this embodiment may be used to support oculars other thandigital loupe oculars. Portions of this head mounting system may also beused to support a single ocular using, e.g., a single ocular support armand associated structure. Display bar 1304 supports the oculars 1305a,b. Display bar 1304 is coupled to distal ends of side support arms1301 a and 1301 b via hinges 1306 a,b to enable the declension angle ofthe oculars to be adjusted. A sliding connection between oculars 1305a,b and display bar 1304 allows adjustment of the oculars to accommodatea range of interpupillary distances, as described above. It should benoted that display bar 1304 as well as the display bars describedpreviously provide additional stability to the oculars by connecting thefull ocular support structure at the bottom as well as the top, i.e., bylinking the distal ends of the support arms, in addition to theirproximal linkages to the head engagement member.

A housing 1390 for a stereo camera pair 1392 a,b is mounted on a centerpiece 1310 of a head engagement member. A distance sensor (not shown)and/or an illumination source (not shown) may also be disposed inhousing 1390, as described with respect to the embodiments above. As inthe embodiment of FIGS. 10A-B, center piece 1310 is designed to stablyengage the user's head in order to support and maintain the stability ofthe stereo camera pair and ocular support subsystems attached to it.Piece 1310 has a toroidal curvature that approximates the curvature ofthe average front of the head. It can include a thin layer of conformalmaterial, such as gel or foam, that rests upon the head, withoutsignificantly decoupling it from motions of the head. Side pieces 1320a,b of the head engagement member connect to center piece 1310 viaflexible couplings (e.g., rotational hinges) as described above. Sidepieces 1320 a,b also have a toroidal curvature that approximates thecurvature of the average head where they would be located on such ahead. They can also include a thin layer of conformal material asdescribed above. These layers of conformal material serve to bettermatch the shape of the wearer's head.

Extending behind housing 1390 is a support arm engagement member 1330mounted onto a linear slide 1321 in order to provide adjustment of aneye relief distance between oculars 1305 a,b and a user's head. Supportarm engagement member 1330 can slide upon linear slide 1321 indirections anterior and posterior with respect to housing 1390. Sidesupport arms 1301 a,b engage with support arm engagement member 1330 viasliders 1332 a,b. Therefore, articulation of linear slide 1321 causes achange in anterior and posterior positioning of the oculars 1305 a,b,and thus eye relief distance, due to their coupling to support armengagement member 1330 through display bar 1304 and side support arms1301 a,b. Support arms 1301 a,b can slide with respect to sliders 1332a,b to enable the effective length of support arms 1301 a,b to beadjusted. The curved proximal sections of support arms 1301 a,b, as wellas the curve of sliders 1332 a,b, follow a circle 1331 (shown in FIG.13C) which has a center point a distance behind the user's eyes. Bysliding the arms 1301 a,b with respect to sliders 1332 a,b, the oculars1305 a,b coupled to the arms 1301 a,b via display bar 1304 also followthis circle, thus enabling adjustment of the height of the oculars 1305a,b with respect to the user's eyes. FIG. 13D shows a differentarticulation state of the positions of support arms 1301 a,b withrespect to support arm engagement member 1130 with a consequently higherposition of oculars 1305 a,b with respect to their positions as depictedin FIG. 13C. FIG. 13D also shows a different articulation state ofsupport arm engagement member 1130 with respect to linear slide 1321 ascompared to its articulation state depicted in FIG. 13C, with aconsequent change in eye relief distance. Loupe declension angle is alsoadjusted into a different state in FIG. 13D by moving display bar 1304about hinges 1306 a,b. When the head engagement member is engaged withthe user's head, sliders 1332 a,b are medial to the user'sfrontotemporales. Together, the display bar 1304 and the support arms1301 a,b extend from their connections to the oculars laterally,posteriorly, and superiorly, then medially toward support arm engagementmember 1330 in positions at or beyond the periphery of the user's fieldof vision. Thus, the ocular support structure of FIGS. 13A-D extendsfrom the oculars to the connection to the head engagement member medialto the user's frontotemporales without extending through a region of theuser's face medial and superior to a center of the user's eyes andinferior to the user's glabella.

FIG. 14 shows a way to couple together the rotational state of two sidesupport arms 1402 a,b of a head-mounted ocular support. Side supportarms 1402 a,b are analogous to arms 715 a,b, and a change in rotationalstate, analogous to the difference between articulation states shown inFIGS. 8B and 8C is contemplated. A change in the rotational state of oneof arms 1402 a,b rotates respectively pulleys 1403 a,b, which sit atoprigid member 1401. Rotation of one of 1403 a,b is transferred to theother of the two in the opposing direction. Here, the mechanism thattransfers the rotational motion is a set of meshing gears 1404 a,bconnected to pulleys 1403 a,b via belts or pushrods. Rotational encodersand motors can also be used to measure the articulation state of oneside arm 1402 a,b and actuate the other to match. This mechanism can beused, e.g., when there is no structure between a pair of oculars (e.g.,the portion of display bar 712 between oculars 711 a,b in FIGS. 7A-C)requiring the oculars to be moved together.

FIG. 15 depicts a support arm structure with ocular supports 1530 a,bsuch that adjusting the declension angle of one ocular supportautomatically adjusts the declension angle of the other to match. Thismechanism can be used when there is no structure between a pair ofoculars (e.g., the portion of display bar 712 between oculars 711 a,b inFIGS. 7A-C) requiring the oculars to be moved together. Part 1501rotationally supports parts 1502 and 1503, and is itself rigidly coupledto the head of the user. Parts 1502 and 1503 remain parallel as they arerotationally connected to linkages 1504 a,b and 1505 a,b. Side supportarms 1510 a,b and 1511 a,b can swivel about linkages 1504 a,b and 1505a,b respectively, and 1520 a,b and 1521 a,b respectively, to adjusttheir clearance with the head of the user. The rotational state of arms1510 a and 1511 a can be coupled through pin 1512 a that mates with aball joint to each arm; similarly for arms 1510 b and 1511 b through pin1512 b. Ocular supports 1530 a,b are rotationally coupled to parts 1520a,b and 1521 a,b respectively, and due to parallel linkages, thedeclension angle of ocular supports 1530 a,b must be the same as parts1502 and 1503, hence adjusting the declension of one ocular results inthe same declension of the other ocular. Alternatively, as above, thedeclension angles of the two oculars can be coupled via asensor/actuator pair.

Each of the articulations described in this disclosure could be manuallyor automatically actuated, for example with a motor. Each may include asensor to determine its state, for feedback and control purposes, orsimply to track usage. As described previously, knowing the relativepositions and orientations of different subsystems of the digital loupesystem, for example, the different declension states of the cameraand/or oculars as well as the distance between them, could enablecompensation of the vertical parallax, or at least the average verticalparallax, that changes as a function of distance away from the surgicalfield.

Additional articulations or articulation ranges not yet described areenvisioned as aspects of the present disclosure. For example, thedisclosure could comprise an articulation or articulation range thatremoves the oculars and/or ocular support structure substantially orcompletely from the user's field of view. This could be done in the caseof digital loupe system 700 of FIGS. 7A-C by articulating hinge 719 suchthat ocular support structure 710 lifts out of the field of vision ofthe user. Similarly, for system 1100 of FIGS. 11A-D, hinge 1103 could bebrought into a state that lifts the oculars 1105 a,b and support arms1101, 1104 completely out of the field of vision. One can envision atrack system like tracks at the ends of arms 1301 a,b, that insert intoslots like 1302 a,b with enough range to lift the oculars and ocularsupport structures completely out of view.

FIGS. 16A-B show a face shield or window that may be used with thedigital loupe system of FIGS. 7A-10B. For clarity, FIGS. 16A-B omit allbut central plate 1010 of the head engagement member of this embodiment.A front face shield plate 1600 cooperates with side face shield plates1602 a and 1602 b to protect the user's face while wearing the digitalloupe system. Side face shield places 1602 a,b are coupled to portionsof side support arms 715 a,b at their top to maintain the freedom toadjust the height of said support arms. Face shield plates 1602 a,barticulate together with side support arms 715 a,b, respectively, toadjust the distance between the face shield plates and the user's facein concert with the same adjustment made to the side support arms 715a,b. As shown, face shield plate 1600 has five facets, including asloped front facet 1604 with a cutout 1606 that permits cameras 720 anddistance sensor 724 to view an object or work area without interferencefrom the face shield. Face shield plate 1600 may connect at the top witha hinge permitting it to be tilted upward. In other embodiments, theface shield may have fewer components or facets, as well as alternativemeans of coupling to ocular support arms and/or head engagementstructures. A face shield may be added to any of the other digital loupesystem or ocular support systems described above.

Digital loupe controls, such as those used for magnification change, orstarting and stopping a video recording, could be actuated via buttonsplaced on the ocular support arms. This is useful because ocular supportarms are easily draped to provide sterility; parts of the ocular supportstructure may already need to be draped to enable the surgeon to adjustvarious articulations intraoperatively. However, articulations that aredriven by motors or other actuators may be commanded to differentpositions in a hands-free manner via voice or gesture or other means ofissuing commands to a digital system.

Placement of digital loupe system components, such as batteries, at theback of the head can be used to counterweight components such as thestereo camera pair and the oculars. The oculars can include built-inheaters, or structures to transfer heat dissipated from displays orother electronics, to keep them warm enough to prevent fogging from theuser's breath.

The processor of the digital loupe system can comprise additionalperipherals that may enhance the system's functionality. For example, itcould comprise a wired or wireless interface for sending video signalsto and from the head-mounted display, such that live video can bestreamed from one digital loupe system to another, or to a server forrecording or streaming to remote locations, or from a server forplayback. A teaching surgeon at a remote location could use such a setupto mark up the field of view of the operating surgeon who may be atrainee, or to telestrate, and indicate points of interest. Variousfunctions may be assisted by the presence of a motion sensing unit suchas an accelerometer, gyroscope, and/or magnetometer.

For purposes of this disclosure, the term “processor” is defined asincluding, but not necessarily being limited to, an instructionexecution system such as a computer/processor based system, anApplication Specific Integrated Circuit (ASIC), a computing device, or ahardware and/or software system that can fetch or obtain the logic froma non-transitory storage medium or a non-transitory computer-readablestorage medium and execute the instructions contained therein.“Processor” can also include any controller, state-machine,microprocessor, cloud-based utility, service or feature, or any otheranalogue, digital and/or mechanical implementation thereof. When afeature or element is herein referred to as being “on” another featureor element, it can be directly on the other feature or element orintervening features and/or elements may also be present. In contrast,when a feature or element is referred to as being “directly on” anotherfeature or element, there are no intervening features or elementspresent. It will also be understood that, when a feature or element isreferred to as being “connected”, “attached” or “coupled” to anotherfeature or element, it can be directly connected, attached or coupled tothe other feature or element or intervening features or elements may bepresent. In contrast, when a feature or element is referred to as being“directly connected”, “directly attached” or “directly coupled” toanother feature or element, there are no intervening features orelements present. Although described or shown with respect to oneembodiment, the features and elements so described or shown can apply toother embodiments. It will also be appreciated by those of skill in theart that references to a structure or feature that is disposed“adjacent” another feature may have portions that overlap or underliethe adjacent feature.

Terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the disclosure.For example, as used herein, the singular forms “a”, “an” and “the” areintended to include the plural forms as well, unless the context clearlyindicates otherwise. It will be further understood that the terms“comprises” and/or “comprising,” when used in this specification,specify the presence of stated features, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, steps, operations, elements, components, and/orgroups thereof. As used herein, the term “and/or” includes any and allcombinations of one or more of the associated listed items and may beabbreviated as “/”.

Spatially relative terms, such as “under”, “below”, “lower”, “over”,“upper” and the like, may be used herein for ease of description todescribe one element or feature's relationship to another element(s) orfeature(s) as illustrated in the figures. It will be understood that thespatially relative terms are intended to encompass differentorientations of the device in use or operation in addition to theorientation depicted in the figures. For example, if a device in thefigures is inverted, elements described as “under” or “beneath” otherelements or features would then be oriented “over” the other elements orfeatures. Thus, the exemplary term “under” can encompass both anorientation of over and under. The device may be otherwise oriented(rotated 90 degrees or at other orientations) and the spatially relativedescriptors used herein interpreted accordingly. Similarly, the terms“upwardly”, “downwardly”, “vertical”, “horizontal” and the like are usedherein for the purpose of explanation only unless specifically indicatedotherwise.

Although the terms “first” and “second” may be used herein to describevarious features/elements (including steps), these features/elementsshould not be limited by these terms, unless the context indicatesotherwise. These terms may be used to distinguish one feature/elementfrom another feature/element. Thus, a first feature/element discussedbelow could be termed a second feature/element, and similarly, a secondfeature/element discussed below could be termed a first feature/elementwithout departing from the teachings of the present disclosure.

Throughout this specification and the claims which follow, unless thecontext requires otherwise, the word “comprise”, and variations such as“comprises” and “comprising” means various components can be co-jointlyemployed in the methods and articles (e.g., compositions and apparatusesincluding device and methods). For example, the term “comprising” willbe understood to imply the inclusion of any stated elements or steps butnot the exclusion of any other elements or steps.

As used herein in the specification and claims, including as used in theexamples and unless otherwise expressly specified, all numbers may beread as if prefaced by the word “about” or “approximately,” even if theterm does not expressly appear. The phrase “about” or “approximately”may be used when describing magnitude and/or position to indicate thatthe value and/or position described is within a reasonable expectedrange of values and/or positions. For example, a numeric value may havea value that is +/−0.1% of the stated value (or range of values), +/−1%of the stated value (or range of values), +/−2% of the stated value (orrange of values), +/−5% of the stated value (or range of values), +/−10%of the stated value (or range of values), etc. Any numerical valuesgiven herein should also be understood to include about or approximatelythat value, unless the context indicates otherwise. For example, if thevalue “10” is disclosed, then “about 10” is also disclosed. Anynumerical range recited herein is intended to include all sub-rangessubsumed therein. It is also understood that when a value is disclosedthat “less than or equal to” the value, “greater than or equal to thevalue” and possible ranges between values are also disclosed, asappropriately understood by the skilled artisan. For example, if thevalue “X” is disclosed the “less than or equal to X” as well as “greaterthan or equal to X” (e.g., where X is a numerical value) is alsodisclosed. It is also understood that the throughout the application,data is provided in a number of different formats, and that this data,represents endpoints and starting points, and ranges for any combinationof the data points. For example, if a particular data point “10” and aparticular data point “15” are disclosed, it is understood that greaterthan, greater than or equal to, less than, less than or equal to, andequal to 10 and 15 are considered disclosed as well as between 10 and15. It is also understood that each unit between two particular unitsare also disclosed. For example, if 10 and 15 are disclosed, then 11,12, 13, and 14 are also disclosed.

Although various illustrative embodiments are described above, any of anumber of changes may be made to various embodiments without departingfrom the scope of the invention as described by the claims. For example,the order in which various described method steps are performed mayoften be changed in alternative embodiments, and in other alternativeembodiments one or more method steps may be skipped altogether. Optionalfeatures of various device and system embodiments may be included insome embodiments and not in others. Therefore, the foregoing descriptionis provided primarily for exemplary purposes and should not beinterpreted to limit the scope of the invention as it is set forth inthe claims.

The examples and illustrations included herein show, by way ofillustration and not of limitation, specific embodiments in which thesubject matter may be practiced. As mentioned, other embodiments may beutilized and derived there from, such that structural and logicalsubstitutions and changes may be made without departing from the scopeof this disclosure. Such embodiments of the inventive subject matter maybe referred to herein individually or collectively by the term“invention” merely for convenience and without intending to voluntarilylimit the scope of this application to any single invention or inventiveconcept, if more than one is, in fact, disclosed. Thus, althoughspecific embodiments have been illustrated and described herein, anyarrangement calculated to achieve the same purpose may be substitutedfor the specific embodiments shown. This disclosure is intended to coverany and all adaptations or variations of various embodiments.Combinations of the above embodiments, and other embodiments notspecifically described herein, will be apparent to those of skill in theart upon reviewing the above description.

What is claimed is:
 1. A method of obtaining a stereoscopic image of anobject, the method comprising: obtaining first and second images of anobject with first and second cameras; obtaining a measurement ofdistance to the object with a distance sensor; and applying atransformation to the first and second images using the measurement ofdistance and using calibration information pertaining to one or more ofa position, an orientation, a focal length and a pixel size of the firstand second cameras, wherein the transformation causes the first andsecond images to appear as if the first and second cameras had virtualoptical axes that converge at a distance corresponding to themeasurement of distance.
 2. The method of claim 1, further comprisingdisplaying the transformed first and second images on first and seconddisplays, respectively.
 3. The method of claim 1 wherein the applyingstep comprises adjusting a field of view of the first and second imagesusing the measurement of distance.
 4. The method of claim 1 furthercomprising using the measurement of distance to shift a viewpoint of thefirst and second images.
 5. The method of claim 1 further comprisingchanging a magnification of the first and second images and adjusting afield of view of the distance sensor with the change of magnification.6. The method of claim 1 further comprising changing the distancebetween the object and the first and second cameras and adjusting thetransformation with the change in distance.
 7. The method of claim 1further comprising illuminating the object.
 8. The method of claim 7further wherein the illuminating step comprises determining anillumination parameter based upon the measurement of distance andilluminating the object based on the illumination parameter.
 9. Themethod of claim 7 wherein the illuminating step comprises pulsing anillumination source in a manner synchronized with exposure intervals ofthe first and second cameras.
 10. The method of claim 1 wherein thefirst and second cameras are mounted on a user's head.
 11. The method ofclaim 1 wherein the distance sensor is mounted on a user's head.
 12. Amethod of viewing an object, comprising: engaging a head engagementmember with a user's head, the head engagement member supporting twocameras above the user's head; placing each of a first display and asecond display in a line of sight with an eye of the user; obtainingfirst and second images of the object with first and second cameras;obtaining a measurement of distance to the object with a distance sensorsupported by the head engagement member; applying a transformation tothe first and second images using the measurement of distance and usingcalibration information pertaining to one or more of a position, anorientation, a focal length and a pixel size of the first and secondcameras, wherein the transformation causes the first and second imagesto appear on the first and second displays as if the first and secondcameras had virtual optical axes that converge at a distancecorresponding to the measurement of distance; and displaying thetransformed first and second images on first and second displays,respectively.
 13. The method of claim 12 further comprising supportingthe first and second displays with the head engagement member.
 14. Themethod of claim 12 wherein the applying step comprises adjusting a fieldof view of the first and second images using the measurement ofdistance.
 15. The method of claim 12 further comprising using themeasurement of distance to shift a viewpoint of the first and secondimages.
 16. The method of claim 12 further comprising changing amagnification of the first and second images and adjusting a field ofview of the distance sensor with the change of magnification.
 17. Themethod of claim 12 further comprising changing the distance between theobject and the first and second cameras and adjusting the transformationwith the change in distance.
 18. The method of claim 12 furthercomprising illuminating the object with an illumination source supportedby the head engagement member.
 19. The method of claim 18 furtherwherein the illuminating step comprises determining an illuminationparameter based upon the measurement of distance and illuminating theobject based on the illumination parameter.
 20. The method of claim 18wherein the illuminating step comprises pulsing an illumination sourcein a manner synchronized with exposure intervals of the first and secondcameras.
 21. The method of claim 10 wherein the distance sensor ismounted on the user's head.